Back to problem
Systems Administrator IAdvancedIncident Response~55-70 min

The Broken Release Server

One down service. Disk pressure reported separately. How many actual problems is that?

Your goal

Your goal is not to find something wrong and fix it. Your goal is to gather evidence before touching anything, work out which defects actually explain today's outage versus which are separate risks, repair each one with the smallest correct change, and verify the whole thing end to end — as the real service account, from more than one working directory, through the actual health endpoint.

Observe
Hypothesize
Test
Repair
Triage
Verify

This one doesn't teach a single new Linux mechanic — it hands you a real incident and asks you to figure out which of your existing skills actually applies. Filesystem paths, permissions, environment, and process all show up here, tangled together the way they really do in production.

01

You get symptoms. Your job is to turn them into evidence.

Here's the entire incident report you're handed:

The whole report
catalog-service is down after a deployment.
Disk usage is also above 90%.
Please investigate.

That's realistic. Real incidents rarely say "the symlink at /opt/catalog/current is wrong." They say a service is down and let you figure out why.

catalog-service is down
the reported symptom
deployment just ran
under investigation
config error?
under investigation
disk at 91%
under investigation

Work this loop instead of guessing:

  • Observe — what actually changed, what's actually broken
  • Hypothesize — a possible, testable explanation ("maybe the deployment points at the wrong release")
  • Test — check it directly, without changing anything yet
  • Evidence — the hypothesis is now supported, or it's ruled out
  • Repair — only once you know what's actually wrong
  • Verify — prove the repair under real conditions
Knowledge check 0

You're told: catalog-service is down after a deployment. Disk usage is also above 90%. Nothing more.

What's the right first move?