Start with the outcome, then rewind the timeline
In an infrastructure support interview, open with the outcome in one sentence: what broke, who was affected, and how you knew it was fixed. Quantify impact: requests per second dropped 70%, p95 latency spiked to 4s, or five regions degraded. State the environment up front (cloud, on‑prem, hybrid), primary components, and the service level objective that was at risk. This primes the interviewer to judge your decisions against the right constraints.
Then rewind the timeline crisply. Use clear markers: T0 detection, T+5 scoping, T+12 hypothesis A, T+20 rollback, T+35 mitigation live. Contrast baseline with deviation, and note what you did to limit blast radius. Call out your MTTD/MTTR and what shortened them. This structure shows executive awareness without losing technical depth, and prevents meandering that hides the real signal: how you think under pressure.
Map the case across three planes
Anchor your explanation to three planes: data plane (packets, processes, storage I/O), control plane (schedulers, routing, service discovery), and management plane (APIs, IAM, observability, CI/CD). Most outages begin with evidence in the data plane, originate in control‑plane drift, and are prolonged by management‑plane blind spots. Stating which plane you probed and why keeps the narrative organized and shows systems thinking.
Example: 502s at the edge (data plane symptom), caused by mismatched timeouts between ingress and service mesh (control plane), undetected due to missing long‑tail latency histogram and noisy alerts (management plane). Explicitly say which tools illuminated each plane—pcap, kubectl, Route 53, ALB logs, Grafana—so interviewers see that your tool choice follows the fault domain, not habit or guesswork.
Use a compact STAR‑TC frame
Compress your story with STAR‑TC: Situation (context and SLO), Trigger (what changed), Analysis (hypotheses and tests), Response (mitigation and fix), Trade‑offs (why this over that), Confirmation (evidence it’s resolved). Keep Situation and Trigger under 30 seconds; spend time on Analysis logic and Trade‑offs. Show at least two competing hypotheses and how you falsified one quickly to conserve time.
Trade‑offs and Confirmation separate seniors from intermediates. Articulate what you did not do and why: avoided cluster‑wide restart due to eviction storms; postponed schema migration. Then prove resolution: error budget burn returned to plan, synthetic tests stable for two hours, no new alerts, customer support volume normalized. Close with prevention: guardrails, monitors, and process changes that reduce recurrence probability or blast radius.
Step-by-step workflow to reconstruct a case
1) Define the SLO and impact. 2) Capture topology: ingress, DNS, load balancers, service mesh, queues, databases, and cross‑region paths. 3) Fix the time window and align all logs to UTC. 4) Form two initial hypotheses from first principles (resource saturation vs path misconfiguration). 5) Test the cheapest falsifiable hypothesis first using one command or query.
6) Isolate fault domain: client, edge, app, data store, or external dependency. 7) Apply a reversible mitigation with a rollback plan and owner. 8) Validate with multiple signals: metrics, traces, logs, and a user‑visible probe. 9) Document exact commands, dashboards, and diffs. 10) Extract preventative controls: alert thresholds, runbooks, config linting, chaos tests, and deployment checks. This workflow shows discipline and reduces reliance on anecdote.
Realistic example: intermittent 502s at the edge
Situation: Global users intermittently hit 502s via AWS ALB -> NGINX ingress -> Kubernetes -> gRPC microservice with sidecar proxy. SLO: 99.9% availability; at T0, error rate was 4% with p95 latency 2.5s. Trigger: last night’s service mesh sidecar update and an HPA policy change. Initial tests: curl with --http2, ALB access logs, kubectl describe ingress, and pcap on an affected node. Symptom sharpened: 502s only on long‑lived gRPC streams over 55 seconds.
Analysis: ALB showed Target.ResponseCode “-” with TargetStatusCode “-”, indicating backend closed early. NGINX had proxy_read_timeout 75s; sidecar keepalive_timeout defaulted to 55s; ALB idle timeout at 60s. Under load, sidecar closed streams at 55s; ALB then surfaced 502. Mitigation: raise sidecar keepalive to 70s, align ALB idle to 120s, set NGINX keepalive_requests to reduce churn, and increase HPA minReplicas to dampen scale‑in. Confirmation: 0.02% errors for four hours, synthetic gRPC test stable, no new 502s in canary region.
Checklist to bring into the room
- One L1, one L2, and one hairy L3 case prepared, each with a 2‑minute and a 7‑minute version. - A one‑slide topology and a 10‑line timeline. - Three key metrics (availability, latency, saturation) and two logs or traces that mattered. - The riskiest decision you made and the rollback you prepared. - One prevention you shipped and one you still want but need buy‑in for.
- Commands you actually ran (e.g., kubectl get endpoints -o wide; aws elbv2 describe-target-health; dig +trace). - The alert that worked and the one that didn’t, with a specific improvement. - Stakeholders you coordinated with (networking, DBAs, support) and how you communicated status. - A 30‑second security note: access controls, approvals, and audit artifacts. Bring these in notes; refer, don’t read.
Common mistakes and what they signal
Vague symptoms (“it was slow”) or outcome‑only stories telegraph limited ownership. Tool worship (nmap everything!) without a falsification plan reads as thrashing. Skipping rollback and safety nets suggests change fatigue or risk blindness. Post‑hoc certainty without explaining dead ends hides your diagnostic process, which is what interviewers actually assess in support and SRE roles.
Overfitting the story to a perfect world is another trap. Real systems have partial rollbacks, noisy alerts, budget limits, and human handoffs. Ignoring the security context—who approved production access, how secrets were handled, what was logged—raises red flags. Tie actions to controls; reference established patterns when relevant, and separate what you validated from what you inferred to avoid overstating confidence.
Next steps, verification sources, and your brief close
Write a 300‑word STAR‑TC script for your best case, then record a 2‑minute and a 7‑minute version. Timebox hypothesis testing in rehearsal to under three minutes. Build a small “evidence folder”: before/after dashboards, one log snippet, and a config diff. Do a red‑team pass: ask a peer to poke holes in your trade‑offs and rollback. Repeat with one network‑centric and one storage‑centric incident to show range.
When you need to align terminology or controls, verify against public frameworks: AWS Well‑Architected for reliability and operations, the NIST Cybersecurity Framework for governance and response, and CISA resources for hardening priorities. If you’re expanding the team or seeking peers to review and raise this bar, Auranik’s Specialist Engineering & Staffing (/poland/technology/engineering-staffing) can help you find practitioners who think and communicate this way.
Community content reflects individual experiences and should not be treated as legal, immigration, financial or government advice.
Know someone who may find this guide useful?