In brief
- AI incidents cover security, privacy, quality, unexpected impact and third-party dependencies.
- The team needs reachable roles, classification criteria and technical stop or containment options in advance.
- Model, prompt, data, guardrail and tool versions are central to investigation.
- Correction, communication, recovery and new regression tests are part of incident closure.
What counts as an AI-related incident?
Not every false sentence is a major incident, but recurring or consequential failures can be. The organisation defines categories and severity according to actual impact rather than only technical cause.
| Severity | Example | Immediate containment | Escalation and restart |
|---|---|---|---|
| Level 1 – quality error | One incorrect internal summary with no onward use or data exposure. | Flag and correct the case and add it to the evaluation set. | Product owner; operation may continue if there is no systemic pattern. |
| Level 2 – control failure | Repeated wrong sources, bypassed guardrail or action beyond a process rule without realised harm. | Constrain the affected function, source or tool route and preserve evidence. | Operations, domain owner and security; restart after a reproduced fix and regression test. |
| Level 3 – major incident | Unauthorised external message, discriminatory effect or disclosure of confidential data. | Stop the action, revoke access, identify affected data and people, and activate incident response. | Security, privacy, leadership and provider; assess legal notification and transparency duties. |
| Level 4 – critical and ongoing | Active data exfiltration, compromised agent or uncontrolled payments and deletions. | Immediately isolate the service or privileged functions, rotate credentials and activate fallback operation. | Crisis management; restart only after formal approval, a closed attack path and strengthened monitoring. |
What happens first?
Containment and evidence preservation must be considered together. A system can be disabled, limited to read-only mode, disconnected from tools or rolled back to a safe version. Relevant logs are protected from alteration at the same time.
- Step 1
Triage
Assess impact, affected people, data and ongoing danger.
- Step 2
Contain
Constrain the function, model route, source, tool or permission precisely.
- Step 3
Preserve
Secure versions, events, approvals and relevant evidence.
- Step 4
Escalate
Engage security, privacy, domain, leadership and provider contacts according to plan.
How are cause and communication handled?
Investigation separates model behaviour, application logic, data, user action and third parties. Legal and privacy owners are involved early when reporting or notification duties may apply. Communication to affected people distinguishes confirmed facts from open questions.
- Reconstruct the timeline and complete data flow
- Identify affected versions, user groups and downstream systems
- Search for similar cases and previously unseen reach
- Assess reporting, notification and contractual duties
- Communicate status, known impact and protective action clearly
When may the system return to operation?
Fixing one example is not enough. The team corrects the underlying control, tests related attack paths and demonstrates that containment works. Named release criteria and enhanced observation apply during recovery.
- Document root cause or an effective compensating control
- Add the failure and variants as regression tests
- Clean up permissions, keys and affected data
- Obtain recovery approval from domain and risk owners
- Close a retrospective with actions, owners and deadlines
Example: unauthorised access through a knowledge assistant
A user reports a citation from a confidential project. The team disables the affected connector, preserves request, permission and index versions and checks for further access. The cause is a delayed group synchronisation. After remediation, role tests are automated, maximum synchronisation delay is monitored and affected people are informed under the established privacy process.
What to remember
Prepare stopping, evidence preservation, roles and communication before an incident. Safe recovery requires demonstrated correction and testing, not hope that the same case will not recur.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed