In brief
- Red teaming creatively searches for weaknesses, while normal evals repeatedly measure known requirements.
- Its scope covers the model, application, identity, data, retrieval, tools, infrastructure and human process.
- Tests need rules, safe environments, authorised targets and responsible handling of discovered data.
- Findings are prioritised, fixed and converted into durable regression tests in the development process.
What does AI red teaming test?
Testing follows the real use case and plausible adversaries. An internal writing tool has different risks from an agent with email and payment access. Unintentional misuse and harmful impact on affected people also belong in scope.
- Prompt injection, jailbreaks and hidden instructions
- Data exfiltration, privilege expansion and unauthorised tool actions
- Manipulated sources, models or dependencies
- Unsafe downstream use of output in code, browsers or business systems
- Circumvention of guardrails, limits, approvals and monitoring
How does a red-team exercise work?
Objectives, system boundaries and success criteria are derived from the threat model before attacks begin. The red team works as independently as practical from implementation, while a blue team controls operations and safety. Dangerous actions are tested in an isolated or simulated environment.
- Step 1
Model threats
Identify valuable data, plausible attackers, impact and trust boundaries.
- Step 2
Set rules of engagement
Document authorised systems, time, data, stop criteria and contacts.
- Step 3
Run attacks
Combine known techniques and creative variants across the entire workflow.
- Step 4
Demonstrate impact
Record a reproducible path, prerequisites and actual consequences.
- Step 5
Retest
Challenge fixes and related bypasses deliberately.
How does the test itself remain safe?
A red team must not lose real customer data or trigger real payments. Production-like systems are useful but need technical boundaries, synthetic accounts, secure evidence storage and an immediately reachable incident contact.
- Written authorisation and clear test scope
- Isolated data, test accounts and constrained tools
- A stop mechanism for unexpected impact
- Protected handling of vulnerabilities and evidence
- Coordinated reporting to internal teams and affected providers
How does red teaming complement other assurance?
Red teaming does not replace functional testing, evals, code review or traditional penetration testing. It discovers new attack ideas and system-level combinations. Reproducible findings should then be automated or maintained as fixed scenarios.
| Assurance method | Core question | Approach and result | Useful timing |
|---|---|---|---|
| Evaluation | Does the solution repeatedly meet known quality and safety requirements? | Fixed test cases and metrics produce comparable results and release gates. | During development, before every release and after model, prompt or data changes. |
| Penetration test | Are there traditional technical vulnerabilities in the application, API, cloud or identity layer? | Targeted technical attacks produce reproducible vulnerabilities and remediation. | Before production and after material architecture or security changes. |
| Audit | Do evidence, processes and promised controls exist and operate effectively? | Documentation, configuration, samples and accountability are reviewed. | At approval, during recurring governance review or provider assessment. |
| AI red teaming | Which unexpected attack chains and bypasses work in the real system? | Adversarial scenarios combine model, data, tools and people; findings become regression tests. | Before higher-risk releases and after new capabilities, data sources or attack patterns. |
| Monitoring | Do known or similar patterns occur in live operation? | Signals, alerts and incident procedures detect deviation and realised impact. | Continuously after approval. |
Example: red teaming a procurement agent
The team creates test suppliers, emails and documents containing direct, hidden and multilingual attacks. The goal is to retrieve internal bids or prepare a bank change without second approval. A finding shows that one tool result is treated as a trusted instruction. The interface is separated, permissions reduced and the scenario added to release evals.
What to remember
Red-team the real use case and its consequences, not only a chat. Turn every demonstrated finding into a fixed control, a regression test and a better monitoring signal.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed