Linux · Systems fluency
How do you troubleshoot a Linux production incident?
Short answer
Define the symptom, gather evidence, and isolate the failing layer before changing anything. Use logs, resource metrics, and recent-change history, then make one controlled change at a time and verify the result.
Start with impact, scope, and timeline. Reproduce the failure if possible, then check what changed: deploys, configuration, certificates, DNS, storage, or traffic. Random restarts destroy evidence and often turn a single failure into an outage.
systemctl status myapp --no-pager
journalctl -u myapp --since "30 min ago" --no-pager
df -h
ss -tulpn | grep 8080Isolate by layer
- Process: is it running, restarting, or blocked? Check systemctl and ps
- Resources: CPU, memory, disk, inodes, and file descriptors
- Network: DNS, route, listener, firewall, and upstream dependency
- Application: logs, latency, error rate, and configuration
- Kernel: dmesg for OOM kills, filesystem errors, and driver problems
Use strace, tcpdump, or perf only when the cheaper evidence is insufficient, and account for their overhead. Preserve logs and metrics before cleanup. After the fix, verify user-visible behavior rather than only the process status.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What is the first thing you do when a service is down?
- 2How do you decide whether to restart or investigate first?
- 3Which logs would you check for an OOM kill?
- 4How do you avoid making an incident worse while debugging?
Say it in the interview
I define impact and scope, then gather evidence before changing anything. I check recent changes, process state, resources, network, and application logs in that order. I isolate one layer at a time and preserve evidence rather than restarting blindly. After the fix I verify user-visible behavior and write follow-up actions.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live Linux mock interview and get scored feedback on depth, clarity and confidence.