Alert triage · Human-in-the-loop AI
soc-casebook: AI triage, human decision, written case
Built a triage pipeline where Claude Sonnet 5.5 drafts the first pass on an alert, a named analyst makes the call, and every alert ends in a written case file.
Three cases mapped to ATT&CK (T1621, T1110.001, T1528), including one where the analyst overrides the AI. CI checks that every committed case file matches its recorded run, so the cases rebuild without an API key. 42 passing pytest tests.
The problem
AI can draft a first pass on an alert fast, but a SOC still needs a named analyst to own the verdict and every containment action, and a record a reviewer can check later.
What I built and owned
The model recommends; the analyst decides
Sonnet 5.5 returns a verdict, confidence, severity, timeline, benign explanations it considered, and recommended actions, all under a fixed JSON schema. Output that refuses, truncates, or does not match the schema fails the run. Nothing executes on the model's word: the analyst sets the final verdict and severity, approves or rejects each action with a reason, and can add actions the model missed.
The override case
A user consented to an app named like the company's sanctioned e-signature vendor, granting mailbox write access. The model saw an unvetted app and stopped at needs more info, low confidence. The analyst compared it against the app catalog: a different app ID, user consent instead of admin consent, and permissions the real vendor was never granted. The analyst called it an illicit consent grant, Critical, and added a tenant-wide grant removal.
Agreement is a decision too
In the MFA fatigue case the analyst agreed with the model but rejected its IP block, because the source was a rotating proxy. In the brute force case the analyst agreed, approved all five recommended actions, and added an account disable.
Auditable without trusting anyone
Enrichment is offline and deterministic. Each recorded run holds the exact system prompt, the input the model saw, its raw output, and the analyst's decision. CI fails if any case file drifts from its run, and gitleaks scans every push. All people, domains, and IPs are fictional, and the alert schema rejects anything outside the documentation ranges.