What Is AI Red Teaming?
A Practical Guide to Testing LLM, RAG and Agent Systems
AI systems now make decisions in production, trigger payments and hold conversations with patient and customer data. So who is stress-testing these systems the way a real attacker would? This guide covers what AI red teaming is, where it diverges from classic penetration testing, which methodologies and tools drive it, and why regulatory pressure — led by the EU AI Act — is turning adversarial testing into a compliance expectation rather than a nice-to-have.
What Is AI Red Teaming?
AI red teaming is an adversarial (offensive) security discipline that sets out to break an AI system — especially a large language model (LLM), a retrieval-augmented generation (RAG) architecture, or an autonomous agent — with the creativity, intent and rigour of a genuine attacker. The objective is to identify, document and prioritise the vulnerabilities that bypass a system's safety controls — prompt injection, jailbreaks, data leakage, excessive agency, model manipulation — before the system ships or a malicious actor gets there first.
The term comes from the military and cybersecurity tradition of the "red team": a group that deliberately adopts the attacker's role to test the defences. In classic cybersecurity, the red team targets the network, the endpoints and the human factor. AI red teaming shifts the attack surface onto the model itself — a component that is programmable in natural language and behaves probabilistically.
The critical distinction is this: in traditional software, the vulnerability lives in the code and is deterministic. In an LLM, the "vulnerability" often lives in the model's behaviour: given the right context and the right words, the model does something it should not do. That is why AI red teaming is not an extension of classic security testing but a separate discipline with its own methodology, toolset and expertise.
How Is AI Red Teaming Different from Classic Penetration Testing?
This is the most common point of confusion. When an organisation says "we already run penetration tests," it usually means the network and application layers. Yet the real attack surface of an LLM-based product sits somewhere those tests never touch: the natural-language dialogue conducted with the model itself.
| Dimension | Classic Penetration Testing | AI Red Teaming |
|---|---|---|
| Target | Network, application, infrastructure, code | Model behaviour (LLM/RAG/agent) |
| Nature of the flaw | Deterministic (SQLi, XSS, RCE) | Probabilistic (behavioural, context-dependent) |
| Reproducibility | Same input → same output | Same input sometimes works, sometimes fails |
| Attack vehicle | Exploit code, payloads, scanners | Natural-language prompt, social manipulation |
| Success criterion | Gaining access or privilege | Policy violation, leakage, unintended action |
| Reference frameworks | OWASP Top 10, PTES, OSSTMM | OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF |
| Re-testing | One-off validation after a patch | Continuous — with every model/prompt update |
The most practical difference hides in reproducibility. In a classic pentest a flaw either exists or it does not; the exploit works and keeps working. In an LLM, the same jailbreak prompt might succeed in two of three attempts and fail in the third. That is the nature of testing a probabilistic system, and it changes how you report: instead of "this vulnerability exists," you say "this attack class works under these conditions with this success rate."
The second difference is what a patch means. You close a classic flaw with a code fix. You "weaken" an LLM behaviour with system-prompt hardening, input/output filters, guardrails or fine-tuning — you rarely eliminate it entirely. That is why AI red teaming is a continuous loop that repeats every time the model is updated, not a one-off audit.
What Is the Attack Surface: LLM, RAG and Agent?
A modern AI product is not a single "model"; it is a stack of interconnected layers, and each layer adds its own attack surface. A red team has to treat those surfaces separately.
1. The LLM layer — the direct prompt surface
The most fundamental surface is the prompt the user sends directly to the model. The primary attack class here is prompt injection: convincing the model to ignore the developer's system instructions and follow the attacker's instructions instead. Jailbreaks (role-play, hypothetical scenarios, gradual escalation), system-prompt leakage and output manipulation all live in this layer.
2. The RAG layer — the indirect injection surface
If the model pulls content from an external knowledge base (a vector database or document store), the attacker does not have to write the prompt directly. They embed the malicious instruction inside a document, a web page or a record that the model will read — this is indirect prompt injection. RAG security covers the integrity of the knowledge base, source isolation, and ensuring retrieved content is never treated as trusted instruction.
3. The agent layer — the action surface
If the system can call tools (sending email, executing code, triggering an API, initiating a payment), the risk multiplies. An injection against a chatbot produces a wrong sentence; an injection against an agent can produce real and irreversible actions. Excessive agency, tool poisoning, memory poisoning and agent-to-agent (A2A) manipulation are the vectors of this layer.
How Does the AI Red Teaming Methodology Work?
A mature AI red teaming engagement is not ad-hoc "trying to trick the model"; it sits on recognised frameworks. These frameworks both guarantee coverage and let findings be reported in a shared language.
OWASP LLM Top 10 (2025) — the vulnerability taxonomy
The OWASP LLM Top 10 standardises the most critical vulnerability categories in LLM applications and serves as a natural checklist for defining a red team's test scope:
| Code | Vulnerability | Red-team focus |
|---|---|---|
| LLM01 | Prompt Injection | Direct/indirect instruction hijacking |
| LLM02 | Sensitive Information Disclosure | PII, system secrets, training-data leakage |
| LLM03 | Supply Chain | Poisoned model/adapter/data source |
| LLM04 | Data & Model Poisoning | Training/fine-tuning data manipulation |
| LLM05 | Improper Output Handling | Output flowing through as XSS/SQLi |
| LLM06 | Excessive Agency | Agent exceeding its permission boundary |
| LLM07 | System Prompt Leakage | Disclosure of hidden instructions/rules |
| LLM08 | Vector & Embedding Weaknesses | RAG source poisoning, embedding inversion |
| LLM09 | Misinformation | Hallucination and trust manipulation |
| LLM10 | Unbounded Consumption | Resource exhaustion, model theft, DoS |
MITRE ATLAS — adversary tactics and techniques
MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is the AI counterpart of MITRE ATT&CK from classic security. It catalogues tactics such as reconnaissance, resource development, initial access, model access, execution, defense evasion, discovery, exfiltration and impact — and, under each tactic, concrete techniques. By mapping the attack chain to ATLAS technique IDs, the red team leaves a trace of "what we tried, under which tactic, and what happened." This lets findings be discussed with the defensive team in a common vocabulary.
NIST AI RMF and AI 600-1 — the governance framework
The NIST AI Risk Management Framework (the Govern-Map-Measure-Manage loop) and the generative-AI-specific NIST AI 600-1 profile position red teaming as a governance activity. Here the red team's output feeds the "Measure" function: identified risks are measured, prioritised and mitigated in the "Manage" phase. This turns a one-shot technical test into a continuous risk-management cycle.
EU AI Act — from good practice to legal expectation
The EU AI Act reframes all of the above as a regulatory obligation rather than an optional exercise. Providers of high-risk AI systems are expected to ensure accuracy, robustness and cybersecurity across the lifecycle, which in practice means adversarial evaluation of the deployed system. Providers of general-purpose AI models — particularly those posing systemic risk — face obligations to perform model evaluation and adversarial testing (red teaming) and to document the results. For any organisation touching the EU market, "we red-teamed the model and here is the evidence" is shifting from a differentiator to a baseline expectation.
Which AI Red Teaming Tools Are Used: PyRIT, garak and Automation?
Human creativity is irreplaceable, but automation frameworks are used to scale coverage and reproducibility. These tools run hundreds of attack probes with different variations against the model and score the results.
| Tool | What it does | Approach |
|---|---|---|
| Microsoft PyRIT | Automated red-teaming framework; attack generation, multi-turn orchestration and scoring | Programmable, LLM-assisted attacker |
| garak | LLM vulnerability scanner; jailbreak, leakage, toxicity and injection probes | Probe library + detectors |
| Giskard / promptfoo | Test suites and regression; catching behavioural drift across updates | Evaluation-focused |
| OWASP GenAI resources | Guidance, threat matrices and testing playbooks | Methodological reference |
The critical limit of automation is this: the probe libraries in these tools are predominantly English and calibrated to global models. Scanning a non-English deployment with only the default garak/PyRIT probes misses a large part of the local attack surface. Automation gives you broad but shallow coverage, while a human red team produces the context-aware, language-specific and creative attacks. A mature engagement includes both.
On the research side, AltaySec runs a Turkish LLM injection dataset and the AltayDuel agent arena precisely to close this gap — systematically collecting the local-language attack patterns that English probe libraries cannot see.
Why Do Language and Local Context Matter?
This is the most overlooked dimension of AI red teaming, and it applies far beyond any single market. Global security vendors design their filters and test tools for English; every other language remains a "bolted-on" afterthought. That gap becomes concrete along three axes, using Turkish as a worked example of a non-English, morphologically rich language operating under a GDPR-equivalent legal regime.
1. The linguistic axis — morphology and code-switching
Turkish is an agglutinative language; a single root can take on dozens of forms as suffixes stack up. A keyword-based safety filter that catches the word "şifre" (password) may miss "şifrelerimizi," "şifreleyerek," or variants with letters or spaces inserted between characters. Turkish–English code-switching (two languages in one sentence) is another blind spot for English-centric filters. A red team must deliberately generate these morphological and code-switched variations — and the same class of failure exists for any language whose structure differs from English.
2. The cultural axis — context-embedded manipulation
Social-manipulation patterns — authority framing, urgency, escalation via politeness — vary by culture. Convincing a corporate assistant with an "the director urgently asked for this" frame is a local vector that does not appear in English probe libraries. Such patterns can only be produced by a team that knows the language and the culture.
3. The legal axis — GDPR-equivalent law and local PII
Local data-protection obligations directly define attack targets. In the EU, GDPR governs personal data; in Turkey, the KVKK (Turkey's GDPR-equivalent data-protection law) does the same. National PII formats — a Turkish national ID number, IBAN or tax number — are patterns that English PII detectors (built around formats like the US SSN) do not catch. Moreover, special-category data under KVKK Article 6 and GDPR Article 9 (health, religious belief, biometric and other sensitive data) form a separate class that most foreign filters do not recognise. A red-team engagement in any given jurisdiction has to specifically target these local PII and special-category leakage vectors.
What Are the AI Red Teaming Process Steps?
A structured engagement typically moves through six phases. These steps adapt the classic pentest lifecycle to the probabilistic nature of AI.
- Scoping and threat modelling: Which system is being tested (LLM/RAG/agent)? Which tools does it have access to? What is the most valuable asset — PII, the system prompt, action privileges? Red lines (running against production, using real data) and rules of engagement are agreed.
- Vulnerability mapping: The OWASP LLM Top 10 classes are brought into scope, MITRE ATLAS tactics are targeted, and product-specific attack scenarios are designed.
- Automated scanning (the broad layer): garak/PyRIT run hundreds of probes to establish a fast, reproducible baseline measurement.
- Manual adversarial testing (the deep layer): A human expert deepens the weak points automation surfaced — applying multi-turn escalation, language-specific bypasses, and context-specific manipulation.
- Validation and scoring: Because behaviour is probabilistic, each finding is repeated to measure a success rate and prioritised by severity, likelihood and impact. Algorithmic validation (for example, checking whether a leaked ID or IBAN is actually valid) eliminates false positives.
- Reporting and re-testing: Findings are mapped to OWASP/ATLAS language, mitigation recommendations (system-prompt hardening, input/output guardrails, human-in-the-loop approval) are provided, and the system is re-tested after mitigation. The loop repeats with every model update.
The fifth and sixth steps are where AI red teaming diverges most from classic testing: instead of a one-shot "pass/fail," you get a continuously measured risk cycle that is re-validated after mitigation. Runtime guardrail layers that provide continuous monitoring are how red-team findings become a live defence in production.
Who Needs AI Red Teaming? (EU AI Act and Beyond)
The short answer: any organisation running an LLM/RAG/agent that faces customers or employees. The longer answer has to do with regulation steadily turning this into a compliance requirement.
- Banking and insurance: Customer chatbots and policy assistants operate on data governed by GDPR-equivalent and financial regulation; leakage and privilege escalation carry direct enforcement risk.
- Healthcare: Special-category health data sits in the highest protection class under both GDPR and KVKK; patient assistants and triage bots should not reach production without adversarial testing.
- Public sector and defence: Citizen-facing AI services carry both data sensitivity and manipulation/disinformation risk.
- E-commerce and telecom: High-volume, automated agent flows are exposed to excessive agency and indirect injection.
On the regulatory side, the direction of travel is clear. The EU AI Act introduces obligations for risk management, robustness testing and adversarial evaluation for high-risk AI systems, and model-evaluation and red-teaming duties for general-purpose AI models with systemic risk — obligations that reach any organisation touching the EU market, wherever it is headquartered. Alongside it, GDPR and GDPR-equivalent national laws (such as Turkey's KVKK) impose existing data-protection duties, and voluntary frameworks like the NIST AI RMF continue to spread. The trend is unmistakable: adversarial testing is moving from "it would be good" to "expected to be documented."
At this point an organisation has two complementary needs: an LLM pentest / red-team engagement that breaks the system before it ships, and a guardrail layer that provides a live defence in production. Red teaming finds the vulnerabilities; the guardrail layer blocks the discovered classes at runtime. They are two sides of the same coin.
Frequently Asked Questions
Is AI red teaming the same thing as jailbreaking?
No. A jailbreak is a single attack technique that bypasses a model's safety rules. AI red teaming is the umbrella discipline that tests dozens of attack classes — including jailbreaks, data leakage, indirect injection, excessive agency and model theft — through a structured methodology. Jailbreaking is just one of the techniques a red team uses.
Are automated tools (garak, PyRIT) enough on their own?
They are not. Automated tools provide broad but shallow baseline coverage, and their probe libraries are predominantly English. Vectors such as multi-turn escalation, context-specific social manipulation and language-specific bypasses require a human expert. A mature engagement combines automation with manual testing.
How often should AI red teaming be done?
The classic annual pentest cadence is insufficient for LLMs. Behaviour can change whenever the model, system prompt, RAG knowledge base or tool set is updated. Red teaming should therefore run on a release-based, continuous basis — ideally supported by runtime monitoring.
Why isn't an English test suite enough for a non-English model?
English probe libraries do not cover the morphology of agglutinative languages, code-switching patterns, culturally specific manipulation frames, or local PII formats such as national IDs governed by GDPR-equivalent law. A model that comes out "clean" in English testing may still be exposed on its local-language attack surface.
Conclusion
AI red teaming is a distinct discipline that stress-tests the layer classic security testing never touches — the model's own behaviour — the way a real attacker would. It is probabilistic rather than deterministic, continuous rather than one-shot, and it must be run with local context rather than English-only templates. A structured approach that combines OWASP LLM Top 10 coverage, the MITRE ATLAS tactic language, the NIST AI RMF governance loop and PyRIT/garak automation produces far more defensible results than "trying to trick the model."
For any organisation operating internationally, the message is twofold. First, local language, cultural manipulation and jurisdiction-specific PII are mandatory parts of scope. Second, with the EU AI Act and adjacent regulation, adversarial testing is evolving from good practice into a documented obligation. At AltaySec we put local-language context at the centre — through research (a Turkish injection dataset, the AltayDuel arena) and in the field, via LLM red-team engagements and the Guardian guardrail layer in production.
Cite this article:
Yurtsevenler, F. E. (2026). What Is AI Red Teaming?
A Practical Guide to Testing LLM, RAG and Agent Systems. AltaySec.
https://altaysec.com.tr/en/arastirmalar/ai-red-teaming-nedir.html