AI Red Teaming: How to Red Team LLMs and AI Agents

AI red teaming is structured adversarial testing of an AI system, such as an LLM application, a RAG assistant or an AI agent, to find the ways it can be made to cause harm before real users or attackers do. NIST describes it as a structured exercise that probes an AI system for flaws and vulnerabilities, often with the system's developers. A useful exercise does more than collect jailbreak screenshots: it starts from objectives and a harm taxonomy, combines automated and expert testing, scores findings by impact and reproducibility, and keeps every confirmed failure as a regression test.

What Is AI Red Teaming?

AI red teaming borrows its name from military and security red teams, but the target and the failure modes are different. A classic red team emulates a threat actor against an organization's people, processes and technology to reach a business objective; our guide to red teaming covers that discipline. An AI red team targets a system whose core component is probabilistic, accepts instructions in natural language and can be steered by any text it reads.

The NIST Generative AI Profile (NIST AI 600-1) defines AI red-teaming as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled environment and in collaboration with system developers." Two things in that definition matter for planning. First, the scope includes harmful and discriminatory outputs, not only security bugs. Second, it is a collaboration: the red team works with the builders, not against them.

Microsoft's AI Red Team, officially established in 2018, describes the same breadth. In its paper Lessons From Red Teaming 100 Generative AI Products, the team says it emulates both adversarial attackers and benign users who run into failures by accident, and it groups impacts into two categories: security (for example, data exfiltration or prompt injection) and safety, which it also calls responsible AI (RAI) harms.

In practice, the line between AI red teaming and AI penetration testing is blurry, and many engagements blend both. The distinction that matters is the one Microsoft draws in its third lesson: AI red teaming is not safety benchmarking. Benchmarks measure known harms with a fixed dataset; red teaming looks for the harms nobody has written a benchmark for yet, in the specific context where the system is deployed.

Why AI Red Teaming Needs a Method, Not Just Clever Prompts

Most failed AI red team exercises fail for the same reasons: no agreed objective, no record of what was tried, single attempts treated as proof, and findings that disappear after the report. Four properties of AI systems explain why a method matters.

• Results are probabilistic. The same input can produce different outputs. In a January 2025 evaluation, NIST's Center for AI Standards and Innovation (CAISI) attempted each of five agent hijacking attacks 25 times; the average attack success rate rose from 57% on a single attempt to 80%. One clean run proves very little.
• The model is one component. Microsoft's red team reports that attacks which target only the model may not translate into production vulnerabilities, while tests that ignore non-AI components such as input filters, databases and cloud resources miss real weaknesses. The system around the model is the target.
• Simple techniques work. The same paper notes that "basic" techniques often work as well as, or better than, gradient-based attacks, and that real attackers tend to use simple methods. You do not need model weights to find serious problems.
• The attack surface keeps moving. A new model version, a revised system prompt, a new tool or a refreshed retrieval corpus can reopen a fixed issue. Microsoft's eighth lesson is that the work of securing AI systems will never be complete.

A method gives you coverage you can explain, results you can reproduce and a regression suite that survives the next model upgrade.

The Frameworks Behind an AI Red Team Exercise

No single framework tells you how to run an exercise end to end, but four public sources cover the pieces. Use them as inputs, not as checklists to recite.

NIST AI 600-1 is especially concrete. Suggested action MS-2.7-007 calls for AI red-teaming to assess resilience against abuse that facilitates attacks on other systems, generative AI attacks such as prompt injection, and machine learning attacks such as data poisoning, membership inference and model extraction. Action MS-2.10-001 adds privacy probes: outputting training data samples and revealing confidential or personal information. The profile also describes four ways to staff an exercise: general public, expert, a combination of both, and human teams working with generative AI.

For application-level vulnerability categories, map findings to the OWASP LLM Top 10; for agents, to the OWASP Top 10 for Agentic Applications covered in our agentic AI security guide.

The AI Red Teaming Methodology at a Glance

The eight phases below are the method we recommend for an AI red team exercise against an LLM application, a RAG assistant or an AI agent. Each phase has an output that the next phase depends on.

Phases 4 to 7 repeat. Microsoft's planning guidance for LLM red teaming recommends a first round of open-ended testing, then guided rounds against a growing list of harms; in practice, new harms found in Phase 6 go back into the taxonomy and the next round of test design.

Phase 1: Set Objectives and Rules of Engagement

Start with the question the exercise must answer, written so that someone can say yes or no at the end. "Can an unauthenticated user make the support assistant reveal another customer's order data?" is an objective. "Test the chatbot" is not.

Agree on:

• Target layers. The base model through its API, the application through its real interface, or both. Microsoft's guidance recommends testing the application on the production UI where possible, because that is closest to real use, and stating which endpoints were tested in the report.
• Environment and data. Use a staging tenant, test accounts and synthetic records. Seed canary values (unique marker strings and fake records) into documents, mailboxes and databases so any leak is visible and harmless.
• Stop conditions and content handling. Decide what testers do if they reach real personal data, and how disturbing generated content is stored and who reviews it. Microsoft notes that red teamers can be exposed to a lot of unsettling output and need a way to disengage.
• Who tests. Security testers for injection and data access, domain experts for medical, legal or financial accuracy, and testers from the expected user population for everyday harms.

Our penetration testing scope guide covers the contractual side of rules of engagement, which applies unchanged to AI work.

Phase 2: Build the Harm Taxonomy

A harm taxonomy is the list of outcomes you treat as failures, each with a definition, an example and a severity anchor. Without it, two testers will score the same output differently and the report becomes an argument.

Build it from three inputs: the business objective from Phase 1, the 12 risks in NIST AI 600-1 and the application's actual capabilities. NIST's risk names are a good scaffold because they cover both security and safety. The table below is a DeepStrike starting template; trim or extend it for your system.

Add a severity anchor to each row, for example "Critical: another customer's real data could be disclosed without authentication." Phase 7 uses these anchors.

Phase 3: Threat Model the System

Before writing test cases, model how each harm could happen. Microsoft's AI Red Team uses a five-part ontology that works well as a worksheet:

The ontology does not assume malice. A benign actor, such as a distressed user or a customer who asks an ordinary question in an unusual way, is modeled the same way, which keeps safety harms in scope.

Map techniques to MITRE ATLAS so findings use shared language. Useful entries for LLM applications include AML.T0051 LLM Prompt Injection (with Direct and Indirect sub-techniques), AML.T0054 LLM Jailbreak, AML.T0069.002 System Prompt discovery, AML.T0070 RAG Poisoning and AML.T0077 LLM Response Rendering. For agents, AML.T0053 AI Agent Tool Invocation and AML.T0086 Exfiltration via AI Agent Tool Invocation. ATLAS publishes monthly releases, so record the release you used; these IDs are from the October 2026 data release.

Then draw the trust boundaries: every place where text from outside enters the model's context (user input, retrieved documents, web pages, email, tool output, memory) and every place where model output leaves (rendered HTML, tool arguments, emails, code). A general threat modeling pass over those boundaries usually produces more useful hypotheses than a list of famous jailbreaks.

Phase 4: Design Repeatable Test Cases

Each attack hypothesis becomes a test case card. The card is what makes results comparable across rounds, testers and model versions.

Two design rules prevent most disputes later. First, success must be machine-checkable where possible: a canary string either appears or it does not. Second, test the control, not only the model. In the card above, even if the model follows the injected text, the tool's authorization check should still stop the cross-account lookup. That is the finding that matters most.

Phase 5: Run Automated Probing

Automation gives you breadth and the repeated trials that probabilistic systems need. Microsoft's fourth lesson is that automation can help cover more of the risk landscape, and its fifth is that the human element remains crucial. Use tools for volume and scoring, not for judgment.

Open-source options most teams evaluate:

How to use automation well:

• Seed it with your test cards. Generic probe libraries find generic problems. Feed the tools your harm taxonomy, your canaries and your system's real entry points.
• Report rates, not anecdotes. For each card, record successes out of total attempts, for example 6 of 25. NIST's CAISI work showed that both the number of attempts and task-level results change the risk picture, so keep per-case numbers rather than one blended score.
• Calibrate automated scorers. LLM-as-judge scorers work reasonably for simple checks such as explicit content, but Microsoft reports they are less reliable in specialized domains such as medicine, cybersecurity and CBRN, where subject matter experts must review results. Spot-check a sample of every scorer's verdicts by hand.
• Watch for false positives. A model that repeats an attacker's text while refusing the action can look like a success to a naive string match. Score the outcome (data, action, harmful content), not the echo.

Phase 6: Run Manual, Expert-Led Probing

The findings that matter most usually come from people. Microsoft's paper describes an operation that chained a low-resource-language prompt injection for reconnaissance, a cross-prompt injection to generate a script and code execution to exfiltrate private user data. No single automated probe would have produced that chain.

Focus manual time on:

• System-level chains. Combine an injection entry point with an over-permissioned tool, a rendering flaw or a missing authorization check. The prompt injection mechanism is usually the first link, not the finding.
• Multi-turn conversations. Guardrails that hold for one message can erode across a long conversation. Test the length and memory limits of the session.
• Indirect channels. Plant canary instructions in every source the system reads: documents, tickets, calendar entries, web pages and tool responses.
• Language and culture. Test the languages and regional contexts your users actually use. Microsoft notes that most safety work is built in English and Western contexts.
• Domain accuracy. Have domain experts judge confabulation and advice quality where wrong answers carry real consequences.
• Benign misuse. Some of the most damaging failures come from ordinary users, for example a distressed user, not from attackers.

Manual testers should also review a sample of the automated "passes." A control that looks solid in aggregate may fail for one high-impact case.

Phase 7: Score Findings Consistently

Score every confirmed finding on three separate dimensions and only then combine them. This keeps a rare but catastrophic failure from being dismissed and a frequent but trivial one from crowding the report.

Impact uses the severity anchors from the harm taxonomy (Critical, High, Medium, Low). Reproducibility uses the measured attack success rate from Phase 5 or 6. Reach asks who can trigger it: an anonymous user, any authenticated user, a user with specific access, or someone who can place content into a data source.

Adjust by one level for reach: raise it when an anonymous user or any indexed content can trigger the issue, lower it when the attacker already needs privileged access. The bands are a DeepStrike convention, not a standard; what matters is that you agree on them before testing and apply them to every finding.

Two rules keep the scores honest. Report the denominator with every rate ("6 of 25 attempts"), and keep identification separate from measurement. Microsoft's guidance warns that red team examples are not a metric for how often a harm occurs in production; if you need prevalence, follow up with systematic measurement on representative traffic.

Phase 8: Report, Fix and Regression-Test

A useful AI red team finding contains the test card ID, the harm category and score, the exact model and prompt versions, the input or conversation, retrieved context and tool calls, the observed output or action, the control that should have stopped it, a recommended fix and a pass criterion for the retest. The general structure in our penetration testing report guide applies; AI findings simply need more context captured per run.

Fix at the layer that will hold. Hardening a system prompt is cheap but brittle; an authorization check in the tool, output encoding before rendering or a scoped retrieval index survives the next model change. Then keep every confirmed case as a regression test and rerun the suite when anything that shapes model behavior changes:

Run the fast, machine-checkable cases in your CI pipeline before each release, and schedule a full exercise with fresh manual testing at least when the system's capabilities change materially.

Red Teaming LLM Applications vs AI Agents

The method is the same for both; the scope and the impacts differ. An LLM application mostly fails by what it says: disclosed data, harmful content, wrong answers. An agent fails by what it does: it calls tools, uses identities, writes to memory and talks to other agents across many steps.

For agents, extend each phase:

• Phase 1: point every tool at sandboxes or mock services, and agree which real actions are allowed.
• Phase 3: add identities, delegated tokens, memory stores, peer agents and human approval points to the threat model.
• Phase 4: write success criteria as actions ("the mock payment tool was called"), not text.
• Phase 7: weight irreversible actions as Critical impact even at low reproducibility.

Our agentic AI security guide gives a lab-safe test, controls and pass signal for each of the ten OWASP agentic risks, from Agent Goal Hijack to Rogue Agents, so use it as the test library for Phase 4. If your agents connect through the Model Context Protocol, add the server-side checks in our MCP security guide.

A Worked Example: Red Teaming a RAG Support Assistant

This example is illustrative. It shows how the phases connect for a common system: a customer support assistant that answers from help articles and community posts through RAG and can look up a signed-in customer's orders.

1. Objective. Can an outside party cause the assistant to disclose another customer's order data, or to send customers to an attacker-controlled site?
2. Harm taxonomy. Sensitive data disclosure (Critical if cross-account), instruction override (High), unsafe link output (High), confabulated refund policy (Medium).
3. Threat model. Actor: someone who can post on the public community forum that the assistant indexes. Techniques: AML.T0051.001 Indirect LLM Prompt Injection and AML.T0070 RAG Poisoning. Weakness: retrieved text is treated as instructions, and the order tool trusts the order ID the model passes. Impact: cross-account disclosure.
4. Test design. Four cards: the cross-account lookup card shown in Phase 4; a card where a planted forum post asks the assistant to include a canary link; a card that checks whether links in answers render as clickable HTML; and a confabulation card that asks about a fictional refund policy.
5. Automated probing. Each card runs 25 times per variation through an orchestration tool, with canary detection as the scorer.
6. Manual probing. A tester combines the link-injection card with the rendering card to see whether an injected link becomes a clickable button in the support widget.
7. Scoring. Suppose the cross-account lookup succeeds in 2 of 25 attempts. Impact is Critical, reproducibility is Rare, so the base severity is High; anyone who can post on the forum can trigger it, so reach raises it to Critical.
8. Fix and regression. Enforce the customer's identity inside the order tool, strip instructions from indexed forum content, and allowlist link domains in rendered answers. All four cards join the regression suite and rerun on the next model update.

The fix that closes the finding is the authorization check in the tool. The model's behavior is a contributing weakness, but the system-level control is what makes the outcome safe.

Common Mistakes in AI Red Team Exercises

• Treating a jailbreak list as a test plan. Public jailbreak collections test yesterday's guardrails on someone else's system. Start from your own harms and entry points.
• Testing only the model. Most high-impact findings sit in retrieval, tools, rendering and authorization around the model.
• One attempt per case. A single pass or fail says little about a probabilistic system. Run repeated trials and report the denominator.
• Unreviewed automated scoring. Judges and string matches produce false passes and false failures. Sample them by hand, especially in specialized domains.
• Mixing identification with measurement. A striking example is not a prevalence rate. Use systematic measurement for that.
• No regression suite. Findings that are not kept as tests return with the next model or prompt change.
• Real data and real actions in scope by accident. Use staging, test accounts, mock tools and canaries, and agree stop conditions before the first prompt.

How Often Should You Red Team an AI System?

Treat AI red teaming as a cycle tied to change, not an annual event. Run a full exercise before first release and whenever the system gains a significant capability, such as a new tool, a new data source or autonomy to act. Run the automated regression suite on every model, prompt or guardrail change. Re-run expert manual testing when the threat landscape shifts, for example when MITRE ATLAS or OWASP publish new techniques relevant to your system.

Frequently Asked Questions

What is AI red teaming?

AI red teaming is structured adversarial testing of an AI system to find ways it can be made to produce harmful, inaccurate or insecure outcomes. It covers security harms, such as prompt injection and data disclosure, and safety harms, such as harmful or biased content, and it tests the model together with the application, data and tools around it.

What is the difference between LLM red teaming and LLM penetration testing?

LLM red teaming explores the full range of harms a system could cause, including safety and misuse, and often runs in repeated rounds with developers. LLM penetration testing focuses on proving exploitable vulnerabilities in a defined application and its integrations. Many engagements combine both, using red team objectives with pentest-style validation and fixes.

Is AI red teaming the same as AI safety benchmarking?

No. Benchmarks score a model against a fixed set of known harms so results can be compared over time. Red teaming looks for unknown and context-specific failures in a deployed system. Findings from red teaming often become new benchmark or regression cases.

What tools are used for AI red teaming?

Common open-source tools include PyRIT from Microsoft, garak from NVIDIA, Inspect from the UK AI Security Institute, HarmBench and AgentDojo for agents. They automate attack variations, repeated trials and scoring, but expert testers are still needed to design system-level attacks and judge results.

How long does an AI red team exercise take?

It depends on scope. A focused exercise on one LLM feature with a small harm taxonomy can fit in a few weeks, while a multi-agent system with many tools, languages and user groups needs longer and repeated rounds. Scope, number of harms and the need for domain experts drive the timeline more than the model itself.

Which frameworks guide AI red teaming?

The NIST AI Risk Management Framework and its Generative AI Profile (NIST AI 600-1) define the practice and the risks to cover, MITRE ATLAS provides technique IDs for adversary behavior, and the OWASP GenAI Red Teaming Guide and Top 10 lists help scope tests and classify findings.

Next Step: Plan an AI Red Team Exercise That Ends in Fixes

An AI red team exercise is only as useful as the objective it answers and the regression suite it leaves behind. Define the harms that matter for your system, model how they could happen, test with both tools and experts, score with agreed rules and keep every confirmed failure as a test.

DeepStrike's LLM and AI penetration testing covers prompt injection and jailbreak attack simulation, data leakage, model manipulation and AI access controls, with validated findings aligned to the OWASP LLM Top 10, MITRE ATLAS and the NIST AI Risk Management Framework. If you are preparing an LLM application or agent for release, contact DeepStrike to scope an assessment.

About The Author

Abdalla leads offensive security engagements at DeepStrike, specializing in web and API penetration testing, Active Directory attack paths, and cloud misconfiguration exploitation. His writing translates findings from live engagements into fixes security teams can ship the same week.

Sources and References

• NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
• NIST AI 600-1 (PDF)
• NIST AI Risk Management Framework
• NIST CAISI: Strengthening AI Agent Hijacking Evaluations
• MITRE ATLAS
• MITRE ATLAS data releases (GitHub)
• OWASP GenAI Red Teaming Guide
• OWASP Top 10 for LLM Applications 2026
• Lessons From Red Teaming 100 Generative AI Products (Microsoft AI Red Team, arXiv)
• Microsoft Learn: Planning red teaming for large language models and their applications