WeaknessASC-2026-0023Major weakness
Jailbreaks: bypassing a model's safety rules
HighWorkarounds only
Role play, false context (claiming to be an authorized tester), many-shot examples and encoding tricks get models to do what their safety training forbids, including writing exploits for an attacker. Published techniques include many-shot jailbreaking (Anthropic), Crescendo and Skeleton Key (Microsoft), Deceptive Delight and Bad Likert Judge (Palo Alto Networks Unit 42), Best-of-N sampling (Hughes et al.) and Policy Puppetry (HiddenLayer); each is under Sources.
How to detect it
- Prompts that claim authority or a testing context to unlock refused behavior
- Long prompts with many fabricated example exchanges
- Output that contains content your use policy prohibits
- Conversations that start harmless and escalate a step at a time toward refused content
- Requests that ask the model to add a warning instead of refusing, or that are dressed up as a policy or configuration file
Workarounds
- Enforce what matters outside the model: permissions, tool scopes, egress and approvals
- Monitor outputs for prohibited content and suspend accounts that repeat jailbreak attempts
- Keep models and their safety classifiers updated
- Run content filters on both the prompt and the output; Unit 42 measured an average drop of 89.2 percentage points in attack success with filtering on
No model is jailbreak-proof; design so a jailbroken model still cannot do serious harm.
How to investigate
- Review the session that produced the output and the account's earlier attempts
- Check whether the output was used (code run, message sent) and contain it
Sources
- Anthropic: Many-shot jailbreaking (April 2024)
- Russinovich, Salem and Eldan (Microsoft): Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack (2024)
- Microsoft: Mitigating Skeleton Key, a new type of generative AI jailbreak technique (June 2024)
- Palo Alto Networks Unit 42: Deceptive Delight: Jailbreak LLMs Through Camouflage and Distraction (October 2024)
- Hughes et al.: Best-of-N Jailbreaking (2024)
- Palo Alto Networks Unit 42: Bad Likert Judge: A Novel Multi-Turn Technique to Jailbreak LLMs by Misusing Their Evaluation Capability (December 2024)
- HiddenLayer: Novel Universal Bypass for All Major LLMs (Policy Puppetry, April 2025)
- OWASP Top 10 for LLM Applications 2025: LLM01 Prompt Injection
- MITRE ATLAS: AML.T0054 LLM Jailbreak