Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Red Teaming

nounAI Evaluation and Reliabilityalso AI red teaming, adversarial testing, red team exercise

In one line

Red teaming is the practice of deliberately attacking or probing a system, in this context an AI model, from an adversary's point of view in order to find weaknesses, harmful behaviors and security flaws before real attackers or users do.

Red teaming is the practice of deliberately attacking or probing a system from an adversary’s point of view in order to find weaknesses before real adversaries do. In artificial intelligence, red teaming means testing a model or an AI-powered product with tricky, hostile or unusual inputs to see whether it produces harmful content, leaks private data, can be manipulated into ignoring its rules or fails in dangerous ways. It is a core method of AI safety work and a major part of AI evaluation, used by developers of large language models, by security teams and increasingly by regulators.

Origins

The idea comes from military and intelligence practice, where a “red team” plays the enemy in exercises and war games to challenge plans and expose blind spots, while the “blue team” defends [1]. Cybersecurity adopted it for penetration testing, in which authorized testers try to break into networks and applications. AI red teaming applies the same mindset to models whose behavior is hard to predict by inspection alone.

What AI red teams look for

  • Harmful content. Instructions for violence, weapons, self-harm or fraud, hate and harassment, sexual content involving minors and other policy-violating material.
  • Jailbreaks. Prompts that trick a model into breaking its safety rules, such as role-play framings, encoded text or multi-step persuasion.
  • Prompt injection. Hidden instructions in web pages, emails or documents that hijack a model acting on a user’s behalf, a serious risk for AI agents.
  • Privacy leakage. Extraction of memorized training data or private information from context; see AI privacy.
  • Bias and unfair treatment. Stereotyped or discriminatory outputs; see algorithmic bias.
  • Misinformation and manipulation. False claims, persuasion and impersonation.
  • Dangerous capabilities. Whether a model gives meaningful uplift to cyberattacks, or to biological or chemical threats, tested by domain experts under controlled conditions.
  • Agent and tool misuse. Whether an agent takes unauthorized or harmful actions, spends money or exposes data.
  • Robustness failures. Odd inputs, typos, other languages and long contexts; see model robustness.
  • Overrefusal. Refusing harmless requests, which is also a failure.

Methods

### Manual red teaming

Skilled people craft attacks using creativity and domain knowledge. Teams may include security experts, social scientists, ethicists, subject-matter specialists and people from affected communities. Human testers excel at novel, context-heavy attacks.

### Automated red teaming

Software generates and scores large numbers of attacks. Perez and colleagues showed that a language model can be prompted or trained to write test inputs that elicit failures from a target model, and a classifier flags harmful responses [2]. Other approaches use search and optimization, such as appended strings found by gradient methods, and attacker models that refine prompts over many turns. Automation gives scale and repeatability, while humans give depth.

### Crowd and public programs

Companies run bug bounties and public challenges where thousands of people try to break a model, and they publish findings.

### Expert and external red teaming

For high-risk capabilities, developers bring in outside specialists under agreement before release, and governments and standards bodies run or require testing.

### System-level testing

The model is only part of the system. Red teams test the full product, including filters, retrieval sources, tools, user interface and account controls.

Process

  1. Define scope and threat model. Who might attack, what they want, which harms matter and what is off limits.
  2. Plan and recruit diverse testers, and set rules of engagement and safety support, since testers may see disturbing content.
  3. Probe across risk categories, languages and modalities such as images and audio.
  4. Record successful and near-miss attacks with exact prompts and context.
  5. Triage by severity, ease and reach.
  6. Mitigate: adjust training, system prompts, filters, tool permissions or product design.
  7. Retest to confirm fixes and check for new problems.
  8. Report and share findings, often in a system card, and add failures to regression tests.

Mitigations that follow

Findings feed into additional safety training, including reinforcement learning from human feedback, refusals and safe completion styles; classifiers that screen inputs and outputs; limits on what tools an agent may use; rate limiting and account monitoring; and clearer user warnings. See AI alignment and LLMOps.

Limits and criticisms

  • Coverage. No test can find every flaw, and the space of possible inputs is vast. Absence of discovered failures is not proof of safety.
  • Moving target. New jailbreaks appear continually, and fixes can introduce new gaps.
  • Subjectivity. What counts as harmful varies by culture and context.
  • Whose perspective. Narrow teams find narrow problems.
  • Wellbeing of testers. Exposure to harmful material needs protections.
  • Red teaming theater. Exercises can be used for reassurance without real follow-up.
  • Information hazards. Reports describing attacks can help attackers, so details are handled carefully.
  • Not a substitute for other safety measures such as design review, monitoring and governance; see AI governance.

Policy context

Governments increasingly mention red teaming in AI policy. Executive actions, safety institutes, the European Union’s AI Act and voluntary commitments by developers refer to adversarial testing for powerful or high-risk models; see AI regulation. Requirements differ by country and change, so check current texts.

Building a good red team

Effective AI red teams are diverse. Security researchers bring attack skills, domain experts judge whether an answer is dangerous or merely sounds alarming, linguists and native speakers test other languages and dialects, and people from affected communities spot harms that others miss. Teams work with clear written rules about what they may try, how to handle sensitive findings and what support is available. They keep detailed logs, so that every successful attack can be reproduced and turned into a regression test. Independence matters: testers who did not build the system, and who are rewarded for finding problems rather than for a clean report, are more likely to find uncomfortable truths.

From findings to fixes

A discovered failure is only the start. Teams ask why it worked. A jailbreak may exploit a gap in training data, a conflict between instructions or a weakness in a filter, and each cause points to a different fix. Quick patches, such as blocking a specific phrase, are brittle, because attackers rephrase. More durable fixes change training, add layers of defense or limit the damage a successful attack can cause, for instance by restricting what an agent can do without confirmation. Fixes are then tested again by the red team and by automated suites, and results are summarized for decision makers who must judge whether residual risk is acceptable.

Red teaming and ongoing monitoring

Because new attacks keep appearing, red teaming does not end at launch. Products collect reports from users and flagged conversations, run periodic exercises and track emerging techniques shared by the security community. Monitoring systems look for signs of misuse in live traffic, subject to privacy rules, and feed what they find back into the test suite; see AI observability. Treating red teaming as a continuing cycle of attack, learning and repair, instead of a one-time gate, reflects how security works in every other field.

FAQs

What is red teaming in AI? It is deliberately probing an AI system with adversarial inputs to find harmful behaviors and security flaws before release.

What is a jailbreak? A prompt that tricks a model into ignoring its safety rules.

What is prompt injection? Hidden instructions placed in content a model reads, which hijack its behavior.

Can red teaming be automated? Yes, models and tools can generate and score attacks at scale, though human experts remain essential.

Does red teaming prove a model is safe? No. It finds problems but cannot guarantee that none remain.

More in AI Evaluation and Reliability

Key facts

  • Red teaming originated in military and intelligence exercises in which a designated team plays the adversary to test plans and defenses, and was later adopted in cybersecurity1.
  • In AI, red teaming tests models for harmful outputs, jailbreaks, bias, privacy leaks and misuse, using both human experts and automated attack tools2.
  • Perez and colleagues showed that language models can be used to generate test cases that find failures in other language models automatically2.

Related concepts

Quick checkWhat is red teaming in AI?Show answer

Red teaming in AI is the practice of deliberately probing an AI system from an adversary's point of view, using crafted prompts and attacks, to find harmful behaviors, security flaws and failures before real attackers or users encounter them.

Sources

  1. Micah Zenko, Red Team: How to Succeed By Thinking Like the Enemy, Basic Books, https://www.basicbooks.com/titles/micah-zenko/red-team/9780465048960/
  2. Ethan Perez et al., Red Teaming Language Models with Language Models, https://arxiv.org/abs/2202.03286

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue