DeepSeek's system prompt, extracted: a jailbreak that talked the model past its own guardrails

What happened

DeepSeek is a Chinese AI lab whose chat models drew intense scrutiny in early 2025. Around January 31, 2025, the Wallarm Security Research Team reported a novel jailbreak that allowed partial-to-complete extraction of DeepSeek's system prompt — the hidden instructions that define the model's behavior and guardrails.

Asked directly for its system prompt, DeepSeek behaves normally and refuses. Wallarm's method instead exploited what it described as the model's "bias-based AI response logic": by steering the model into responding under certain biases, they induced it to break some of its internal controls and surface the concealed instructions. Wallarm withheld the exact technique under responsible-disclosure practice and reported that DeepSeek subsequently fixed the issue. The firm also noted that analysis of the extracted material fed into a separate debate about whether DeepSeek's training leaned on other models' outputs. The core security lesson stands regardless: a guardrail the vendor assumed was hidden was recoverable through conversation alone.

Root cause

The system prompt was treated as a secret, but it shares a channel with untrusted user input. In an LLM, system instructions and user turns are ultimately the same token stream; a sufficiently crafted conversation can pressure the model into revealing or overriding its own instructions. Relying on the model to keep its prompt hidden is relying on the model to perfectly resist manipulation — which it does not. This is OWASP LLM07 System Prompt Leakage, new to the 2025 Top 10.

How it would have been caught

A red-team suite of system-prompt-extraction probes (role-play framings, bias induction, indirect elicitation) run against the deployed model, asserting that no probe reproduces the hidden instructions, would have caught the exposure before it was public. Tools such as Garak and PyRIT include leakage tests of this kind.

How to prevent it

• Assume the system prompt is discoverable — never place secrets, API keys, or security-critical logic inside it.
• Enforce policy in code outside the model (authorization checks, output filters), not through prompt instructions the model can be talked past.
• Add an output guard that detects and blocks responses reproducing system-instruction content.
• Continuously red-team for prompt-extraction, since each model update can regress.

The Breachwire test (red → green)

Run a battery of extraction jailbreaks against the model and confirm at least one reproduces the hidden system prompt (RED — the guardrail leaks). Move all secrets and enforcement out of the prompt into code, add an output guard, then confirm the probes no longer surface any confidential instruction while normal requests still get correct answers.