LearnGrokUse the same evidence pack after every meaningful incident or recurring alert. You will leave with a ranked list of runbook changes, named owners, missing diagnostic steps and one next decision for each item.
This is for on-call leads and platform teams whose runbooks exist but do not reliably help someone during an alert. The aim is not to rewrite every page. It is to fix the gaps that cost the next responder time.
Key point
Review the evidence, not the incident summary
A resolved incident can still expose a runbook that sent responders down the wrong path or omitted the check that mattered.
1. Set a fixed review boundary
Run this review after a significant incident, after a repeated alert, or at the end of an on-call rotation. Use a consistent time range, such as the completed rotation, so the team does not select only memorable failures.
Gather these items for each event:
- The incident record, including timeline, impact, mitigation and follow-up actions.
- Alert history, including alert name, firing time, acknowledgement time, repetitions and resolution time.
- The relevant handover notes, especially unresolved risks and workarounds.
- The runbook used, or the runbook that responders expected to find.
- Links or pasted excerpts from investigation notes, dashboards and tickets used during the response.
Remove customer data, secrets, access tokens and unnecessary personal information before sharing the pack. Keep alert names, service names, timestamps and failed commands or queries where they explain the response path.
If your source material is too large to review together, split it by incident or service. The amount you can submit is version-dependent. Check the xAI documentation overview before deciding how to divide a large pack.
2. Use one evidence format
Make a short record for each incident before asking for analysis. This stops a polished narrative from hiding missing facts.
Use this template:
Incident ID:
Service:
Alert or trigger:
User impact:
Start and end time:
Runbook consulted:
What responders did first:
What produced useful evidence:
What did not help or delayed response:
Mitigation or recovery action:
Unresolved question:
Handover item:
Do not fill gaps with guesses. Write not recorded where the timeline does not establish something. That phrase is useful evidence: it may indicate that the runbook needs a recording step, not merely another troubleshooting step.
Watch out
Do not merge separate failures because the alert name matches
The same alert can arise from different causes. Keep records separate until the evidence shows that the diagnostic path and corrective action are genuinely the same.
3. Ask for a ranked update list
Paste the incident records, alert excerpts and handover notes below a prompt like this. Keep the requested output structure unchanged across reviews, so you can compare one rotation with the next.
You are reviewing operational evidence to improve runbooks.
For each proposed runbook update, identify the evidence supporting it. Do not infer facts that are not in the records. Treat `not recorded` as a documentation gap where appropriate.
Return a ranked table with these columns:
1. Rank
2. Runbook or service
3. Observed gap
4. Missing or incorrect diagnostic step
5. Evidence from incident, alert, or handover
6. Proposed change, written as an imperative runbook step
7. Suggested owner role
8. Next operational decision
9. Confidence: high, medium, or low
Rank first by risk of repeated user impact, then by responder time lost, then by frequency. Mark items low confidence when evidence is incomplete or conflicting.
After the table, provide:
- Duplicate or overlapping updates to merge
- Questions that must be answered before editing a runbook
- Alerts that need a separate alert-quality review rather than a runbook change
Evidence pack:
[PASTE RECORDS HERE]
The key instruction is to separate an observed gap from a proposed fix. “The team took 20 minutes to locate the dashboard” is an observed gap. “Add the dashboard link under Initial checks” is a proposed change. Keeping them separate lets you challenge the fix without losing the underlying evidence.
4. Turn each row into an owned change
Review the ranked table with the service owner or the person who carries the relevant operational responsibility. Do not assign every item to the documentation maintainer. A missing rollback check may need an application owner. A misleading alert may need the platform or observability team.
For each accepted row, record these fields in your runbook work queue:
| Field | What to enter |
|---|---|
| Runbook location | The page or repository path to edit |
| Change owner | A named role or team, not on-call |
| Reviewer | The person accountable for technical accuracy |
| Acceptance check | The test showing a responder can use the new step |
| Next decision | Approve edit, gather evidence, change alert, or retire runbook |
Write the proposed change as an action a responder can carry out. Prefer Check error rate for the affected region before restarting workers over Investigate regional errors. Name the expected signal and what choice it informs.
Check
A runbook step is usable when it changes a decision
Ask: “After completing this step, what do I choose next?” If the answer is unclear, the step is reference material, not an operational instruction.
5. Check the analysis before editing
Treat the ranked list as a review draft, not as a source of truth. Check the highest-ranked items against the original incident material first. This is where teams prevent a plausible but harmful runbook change.
Look for these failure patterns:
| If you see this | It may mean | Do this |
|---|---|---|
| A claimed cause has no timeline evidence | The analysis filled in a likely story | Mark it unconfirmed and request the missing record |
| Several incidents produce one generic update | Different failure modes were collapsed | Split the item by trigger or service behaviour |
| A proposed step says “check logs” | The step lacks a decision point | Specify the log source, search condition and next branch |
| An item has no owner | Nobody can validate the change | Assign the responsible team before prioritising it |
| The same alert repeats after documented recovery | Alert behaviour may be the problem | Send it to alert-quality review, not the runbook queue |
Read the proposed change as if you were the responder at 03:00 with no background context. Check that links point to the right service, permissions are realistic, and the order is safe. Confirm that a step does not ask a responder to make an irreversible production change before collecting the evidence needed to justify it.
Stop
Do not publish an unverified diagnostic path after an incident
A fast edit that sends the next responder to the wrong system is worse than a visible gap. Keep uncertain changes in review until the service owner confirms them.
6. Make this a rotation-close habit
Reserve a short review at handover or rotation close. The outgoing lead prepares the evidence pack. The incoming lead checks the top items and confirms that ownership is real. This divides the work without asking the current responder to reconstruct an event weeks later.
Keep a small change log in each runbook: incident reference, date reviewed, change owner, and the reason for the edit. On the next related alert, ask the responder one question: did the added step shorten the path to the next decision? Record the answer in the new handover note.
Do not measure success by the number of pages edited. A good cycle removes a repeated delay, makes an escalation decision clearer, or identifies that the alert itself should change.
When the process does not work
If the output is generic, reduce the evidence pack to one incident and include the actual actions responders took. If rankings feel wrong, state your local priority order explicitly, such as safety, customer impact, recovery time and recurrence. If ownership is repeatedly missing, make the service ownership map a required attachment to the review.
If records cannot support a proposed update, leave it as a question with an owner. The correct next action may be better incident recording or alert review, not another runbook paragraph.
