The Missing Configuration
The inventory-api service expects its configuration at /etc/inventory/app.conf. After a cleanup, startup fails with "configuration file not found."
Works as Root
The watchtower monitoring service starts with sudo but cannot read its rules file when run as the real service account.
The Wrong Binary
The deploy user invokes an obsolete db-backup even though the supported version is installed and works by absolute path.
Where Did the Logs Go?
Security's failed-login report returns zero because its shell pipeline does not match the real authentication log format.
The Broken Release Server
A catalog-service deployment completed, but the service never returned and the host is also under severe disk pressure.
The New Engineer Is Locked Out
Maya cannot read the ops runbooks available to existing operators.
The Shared Reports Directory
Finance analysts cannot reliably edit one another's newly created reports.
The Intern Has Too Much Sudo
An intern received unrestricted passwordless sudo for a two-command support task.
The Permission Bits Look Right
An ACL mask blocks developer write access while a contractor retains an unintended exception.
The Access-Control Incident
Payroll job, finance, contractor, and support access have drifted away from the intended policy.
The Runaway Worker
One batch worker consumes CPU while a healthy same-name worker must remain running.
The Service Won't Start
report-api fails because its loaded unit points to an obsolete executable.
Works Manually, Fails in systemd
notifier depends on an interactive token and PATH absent from its service context.
The Restart Loop
An invalid cache directory and aggressive restart policy create a rapid crash loop.
The Web Stack Incident
nginx returns 502 because orders-api races a restarting mock database.
The Disk Is Full, But Where?
A build server reports "No space left on device." and / is nearly full, but the obvious application directory is small.
The New Disk Exists but the App Cannot Use It
A new virtual disk was attached for /srv/archive. lsblk sees it, but /srv/archive still uses the root filesystem and fills it.
The Filesystem Became Read-Only
orders-writer's writes begin failing with "Read-only file system." Permissions look correct; the filesystem was remounted read-only after a simulated metadata error.
The Backup Filled the Wrong Disk
A nightly backup script writes to /mnt/backup even when the backup volume is not mounted, silently filling / instead.
Storage Incident: Archive Service Down
archive-api is failing uploads. Monitoring says disk pressure, and a new volume was recently added -- but a wrong mount and an inode-heavy temp tree are both involved.
The Service Is Listening, But Nobody Can Reach It
metrics-api works with curl localhost:9100 on the server but remote clients cannot connect. The service is bound only to 127.0.0.1.
The Hostname Works Here but Not There
billing-api must call db.internal.example. One host resolves it; this one doesn't -- a stale /etc/hosts entry and resolver assumptions complicate diagnosis.
Connection Refused, Timeout, or No Route?
A deployment health check cannot reach config-api. Different attempts produce connection refused and timeout depending on the target, and treating every failure as "network down" won't find the real cause.
The Firewall Rule Broke Production
After hardening, web-api is healthy locally but external health checks fail. SSH must remain reachable, and only TCP 8443 should be opened for the app.
Network Incident: API Cannot Reach Database
checkout-api returns 500 because it cannot reach orders-db. A stale DNS override, a database bound to loopback, and a firewall distractor are all present -- the learner must find the actual chain and avoid unnecessary changes.
Find the Error in 2 GB of Logs
A worker failed sometime between 14:00 and 14:10. Its log is huge. Find the relevant failure and produce a concise evidence file without reading everything manually.
The Job Never Ran
cleanup.sh works manually but the nightly cleanup never produces output. The cron entry uses a relative command and assumes the interactive PATH.
The Update Broke the Service
agent-service stopped after a package update. The repository contains the previous known-good version. Restore service while preserving config.
Logs Ate the Server
api-access.log grew huge because rotation is broken. Free space is low, but the service cannot be stopped for long.
Operations Incident: Overnight Automation Failed
Morning operations reports that backups did not run, the monitoring agent is down, and /var is nearly full. A package update, broken rotation, and a cron environment issue are all present at once -- prioritize and restore the host safely.
The New Engineer Can Log In but Cannot Do the Job
Maya can log in and was added to the deployment team, but gets Permission denied entering /srv/deploy while existing engineers work there fine.
Sudo Works for Everyone Except the On-Call Command
A malformed sudoers replacement rule denies the on-call engineer's required inventory.service restart after a broad legacy grant was removed.
The Account Exists, but Login Is Rejected
A returning contractor's account still exists with correct shell and home, but authentication is rejected -- a password lock or account expiry from a past offboarding.
The Application Runs as Root for No Good Reason
inventory-web's systemd unit runs as root, though it only needs to read a config file and write its own state directory.
Access Review: Remove the Privilege Without Breaking Production
Alex changed roles, but deployment privilege persists through group membership, a sudoers drop-in, and a named ACL entry all at once.
SSH Key Authentication Suddenly Stopped Working
Key login stopped working after a home-directory migration; the key is still present, but sshd logs point to unsafe .ssh ownership and modes.
sshd Won't Reload After a Security Change
A hardening edit to sshd_config has an invalid directive or value, blocking reload -- though the current SSH session is still alive.
The Right Key Is Installed, but It Grants Too Much
One automation public key is trusted by both a human account and deploybot, which per policy should only run one safe deployment command.
Users Can Reach SSH From Everywhere
sshd and key authentication are correct, but host firewall policy currently allows TCP/22 from all sources instead of just the management subnet.
SSH Access Incident: Diagnose the Whole Path
After maintenance, Maya's SSH key stopped working while another engineer's still works, even though Maya reaches the host and gets the SSH banner.
The Backup Exists, but It Cannot Restore the Application
A recent nightly archive exists, but a restore drill fails because /etc/inventory configuration was never included in its scope.
The rsync Backup Deleted the Wrong Things
A mirror job using rsync --delete had a source-path/trailing-slash mistake that removed files outside the intended mirror.
The Restore Changed Ownership and the Service Won't Start
A server rebuild restored inventory's files correctly, but ownership/mode metadata on the restored state is wrong and the app fails with Permission denied.
The Nightly Backup Script Works Manually but Fails Automatically
A backup script succeeds when an admin runs it by hand, but the systemd timer invoking it fails due to PATH and cwd assumptions and poor exit-status handling.
Recovery Day: Rebuild the Service From Backups
A simulated disk failure removed inventory's data and config. A recent data archive and an older config archive exist, with one documented config change in between.
The Server Is at 100% CPU
The api-worker service is pegging a full CPU core while an unrelated billing-sync service runs normally alongside it.
Memory Is Disappearing
Available memory keeps shrinking on the host because the cache-worker service holds on to memory it should be releasing between refresh cycles.
Disk I/O Is Making Everything Slow
app-service's /health endpoint is timing out or responding very slowly -- another process on the same host is hammering local disk with heavy, continuous writes.
Too Many Processes, Too Few Resources
The host is sluggish and ps -ef shows hundreds of near-identical worker processes piling up under web-pool.service.
Diagnose the Bottleneck Before You Touch Anything
app-service's health check is failing intermittently while the server shows several signals at once -- moderate CPU noise, journal warnings, and a nearly-full working directory. Only one is the real cause.
The Service Is Running, but Users Cannot Reach It
orders-api shows systemctl status as active (running) and answers curl on localhost:8090, but users elsewhere on the network can't reach it at all.
It Works by IP but Fails by Name
billing-api answers fine when you curl its container's real IP directly, but curl by hostname fails -- even though DNS itself has the right answer.
The Deployment Broke File Access
orders-app runs under its own service account and crashes immediately after the latest deploy: the deploy tooling recreated its data directory with the wrong ownership and mode.
The Nightly Job Says Success But No Backup Exists
A nightly cron job is supposed to archive orders.csv and its log says it's succeeding, but there's no backup in /var/backups/orders -- it was written and tested from an interactive shell.
Final Incident: Recover the Host Safely
Track capstone: inventory-service is down because a misconfigured maintenance job pointed its cleanup target at the service's own work directory, filling it with real bulk files past its startup capacity check.