Choreography: incident response with a copilot
AI correlates logs across services and puts together the incident timeline in minutes, but the root cause, the rollback, and the postmortem's signature stay with the human incident commander.
Three in the morning, PagerDuty fires off. Checkout is returning a 500 error for 12% of orders, and the whole team wakes up trying to understand what changed in the last two hours. Before, that meant scouring Datadog, Grafana, and the payment service log one by one, almost forty minutes just to build the timeline before even considering a cause. Today you paste the three services' logs into AI and, in under two minutes, it returns a timeline with the time of each error, the deploy that shipped at 2:48 AM, and the previous stable version. It also suggests, with the same confidence as always, that the cause is a database migration that ran alongside it. Except that migration had already run the week before with no problem at all, and it's you, the incident commander, who needs to notice that before deciding on the rollback.
A customer threatens to terminate a supply contract alleging breach of the contractual SLA, and you have 48 hours to put together the defense. You ask AI to cross-reference delivery history with the contract text, and it confidently points out the contractual SLA was 95% and you delivered 93%, favoring the customer. The real contract defined 90% as the target, not 95%, a number AI pulled from a generic model contract, not the actual signed document. Checking the original clause before accepting that reading avoids conceding a point that legally doesn't exist.
The sales team's climate survey dropped 15 points on the recognition axis, and HR wants an explanation by Friday. You ask AI to cross-reference the survey with turnover and promotion data from the last two quarters, and it quickly builds an event timeline and points, confidently, to the promotion freeze in March as the cause. Promotions weren't frozen, the process just got slower because of a new calibration committee. It's you, looking at the process up close, who notices the difference before announcing a wrong cause to the whole team.
D7 retention for a new feature drops 25% on some random Tuesday, with no change announcement at all. You ask AI to cross-reference product events with the week's deploy changelog, and it quickly builds the timeline, pointing to an A/B test as the likely cause, with the same confidence as always. The experiment really did run that day, but it affected only 5% of users, not the 25% that dropped. The real cause was a caching bug nobody had connected to the experiment. It's on you to cross-check the percentage before closing the cause and writing the report for the product team.
This month's pipeline drops 18% in a week, and the sales director wants to know why before the forecast meeting. You ask AI to cross-reference the CRM with the holiday calendar and seasonality history, and it confidently points out the drop is normal end-of-quarter seasonality. Except, looking closely, three salespeople left the team in the last two weeks, and that detail is what really explains the drop, not seasonality. It's you who cross-references both signals before bringing a wrong cause to the meeting that decides next month's target.
Delivery SLA drops from 96% to 81% in a single week, and the Zendesk dashboard shows complaints climbing fast. You ask AI to cross-reference the time of each operational step with the volume processed, and it points, with total confidence, to the main distribution center as the bottleneck. The distribution center was normal; the delay came from a new carrier, hired ten days ago, whose route wasn't calibrated yet. Confirming this in the tracking system, before blaming the wrong center, is your job.
An automatic compliance report flags an LGPD control as failed, days before an external audit. You ask AI to investigate the origin, cross-referencing access logs with the current policy, and it confidently concludes the problem is missing recorded consent. Consent was recorded, just under a field with a different name than the one the check script expected. It's you, opening the original record, who avoids reporting an LGPD incident that doesn't exist to the risk committee.
The signup flow's completion rate drops 12 points after a redesign, and the product team wants to understand why before the next sprint. You ask AI to cross-reference the recorded sessions with the flow's changelog, and it confidently points to the password field as the abandonment point. The sessions show abandonment actually happening at the previous step, email verification, which nobody had flagged as suspicious. Reviewing the sessions closely, instead of accepting the first guess, is what keeps you from redesigning the wrong field.
Tuesday morning, the board asks why the Southern market expansion went 30% over budget, and you have until the afternoon to explain. You throw the business unit's financial reports into AI, which builds a month-by-month spending timeline in minutes and points out, with total confidence, that the overrun came from paid media cost. Paid media cost, in fact, stayed within budget; the overrun came from a hired consultancy AI didn't properly separate out in the spreadsheet. Before taking that cause to the board, you're the one who checks it line by line and corrects the reading, because a wrong cause at this level becomes a wrong decision about the entire unit.
Whoa, three in the morning, alarm going off, and the first question in your head is never "what looks nice", it's "what changed". Before, that meant opening three, four different dashboards and cross-referencing logs by hand, half an hour lost just understanding the crime scene before even considering a cause. AI changes this part in a real way: it reads all of that fast and hands you a timeline. The problem is it also points to a cause, with the same confident look as always, and it hasn't always been right.
The core idea of this lesson. An incident has a choreography with fixed steps, and AI comes in strong on the mechanical steps: correlating logs across services, building the timeline, drafting the postmortem. What stays yours, always, is the root cause, the decision to rollback, fix, or wait, the decision to escalate, and the postmortem's final signature. AI lights up the scene; you read the scene and decide what to do with it. And like everywhere in this module, the cause it proposes is suspect until you check it against the real log.
01The incident choreography, step by step
A well-run incident has an order, and skipping a step is what makes a good team get a simple decision wrong under pressure. Detect, build the timeline, raise the likely cause, decide the action, communicate, and finally write the postmortem. Six steps, well-defined roles between machine and person.
AI comes in strong on steps two and six: building the timeline from raw logs, and drafting the postmortem from what already happened. You come in strong on steps three, four, and five: raising the real cause, deciding the action, and communicating in your own voice to whoever needs to know. Step one, detecting, usually already comes from an automatic alert, PagerDuty, Datadog, whatever it is; the choreography really begins when someone, you, takes command.
02Where AI comes in strong: correlating logs and building the timeline
Here AI really earns its keep. A modern incident crosses services: the frontend, the API, the database, an external vendor, each with its own log, in different tools. Piecing that together by hand, looking at timestamp after timestamp in Datadog, Grafana, and the raw service log, is the kind of mechanical work that eats the first precious twenty, thirty minutes of any incident.
Paste the relevant logs into AI and ask for exactly this: a timeline with the time of each event, what changed near the start of the problem (deploy, migration, config change), and the affected services in order. It does this in seconds, and the time gain is real. But notice the limit: this is event reading, not a conclusion about the cause. AI points to what happened and when. What that means isn't decided yet.
03Where it doesn't come in: root cause, rollback, and escalation
Here lies the risk that names this lesson. After building the timeline, most AI tools go one step further and suggest a likely cause. And it does so with the same confident tone as always, whether the cause is right or invented. It might cite the wrong line of the diff, blame a service that only coincided in time, or repeat a common failure pattern that isn't your case.
That's why the real root cause is always your decision, checked against the original log, not accepted because "it sounds reasonable". The same goes for the three decisions that follow: doing a rollback, applying a hotfix, or waiting; escalating to another team or not; and declaring the incident's severity. None of these decisions is purely technical, all carry a judgment of risk and context only whoever's in command of the incident can weigh. AI never takes command. It delivers ammunition for the commander to decide.
04The living runbook
Every incident that gets resolved carries a lesson, and most of those lessons die in the team's chat instead of becoming reusable knowledge. The living runbook is the opposite of that: every real incident that crosses your desk becomes a concrete update to the runbook, not a quarterly review nobody remembers to do.
After closing an incident, ask AI to compare what happened with the current runbook: was this scenario already covered? Does the step that worked need to become a new procedure? This is the same principle of the self-improving system you already saw in the Financial module, applied here: the runbook isn't a document you write once, it's a piece that grows with every real delivery. You already saw, in lesson 6.1, AI operating an entire process with quality gates; the living runbook is that same process's knowledge gate, getting stronger with every round.
Learn more: why the blameless postmortem protects the quality of the root cause
The blameless postmortem practice exists because, when someone fears being pointed at as the mistake, the tendency is to simplify the root cause to close the matter quickly, "it was a configuration error" instead of investigating why the process let that error reach production. This matters twice as much when AI is in the mix: if the team accepts the first cause it suggests just to close the case faster, the real cause, usually a process failure, stays hidden and the same incident happens again. A carefully audited blameless postmortem is what keeps both the person and the machine from becoming a cheap scapegoat for a cause nobody properly investigated.
05Auditing the postmortem before publishing
The postmortem draft AI writes is usually good: it organizes the timeline, describes the impact, suggests prevention actions. The risk lives exactly in the root cause section, because that's where it can invent a plausible explanation that doesn't match the real log.
Before publishing, whoever signs the postmortem checks the root cause line by line against the original log, the same audit principle running through this entire module since the first lesson. If the cause doesn't match, the postmortem doesn't go out, no matter how well written it is. A postmortem published with the wrong cause isn't just a record error, it's a wrong lesson the whole team will carry into the next similar incident.
Do it now
Pick a recent real incident from your team, your real task or another one, or simulate a similar scenario if you don't have one handy.
- Gather the relevant logs (from the affected service and at least one neighboring service) and ask AI to build the timeline: what happened, in what order, what changed near the start of the problem.
- Ask it to suggest a likely cause and draft the postmortem from that timeline.
- Now do your part: open the original log and check, line by line, whether the cause it suggested really matches what happened. Note whether it matched or not.
- If it didn't match, write the real cause and adjust the postmortem before considering it ready for anyone to read.
You've just run the whole choreography, and the part that separated a trustworthy postmortem from a pretty, wrong one was exactly the check you did in step three.
Practice
1. In the incident response choreography with an AI copilot, what does it do well and with strength?
2. During an incident, AI suggested the root cause was a database migration that had already run without a problem the previous week. What's the correct attitude?
3. What makes a runbook 'living', in this lesson's sense?
For the board
On the first questionat three in the morning nobody asks what looks nice. They ask what changed.
On the splitbuilding the timeline is the AI's job. Root cause, rollback and escalation stay with the team.
On the runbookit grows with every real incident, absorbing what worked and what was missing.
Thanks for the feedback. It helps sharpen the next lesson.