PoisonedRAG: five planted passages in a million-document corpus flip a RAG system's answer

What happened

Retrieval-augmented generation (RAG) grounds an LLM's answers in an external knowledge base: a retriever pulls the most relevant passages for a question and the model answers from them. PoisonedRAG, by Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia (USENIX Security Symposium 2025), demonstrated a knowledge-corruption attack against exactly this design.

The attacker injects a small number of crafted malicious texts into the knowledge database. Each text is engineered to satisfy two conditions at once: it is retrievable for a specific target question (so it lands in the model's context), and it carries content that steers the model toward an attacker-chosen answer. The headline result: injecting about five malicious texts per target question into a knowledge base containing millions of texts yields roughly a 90% attack success rate — the model returns the attacker's target answer for that question. The malicious fraction of the corpus is vanishingly small, which is what makes the attack practical and hard to notice.

Root cause

The retrieval corpus is a trust boundary, but RAG systems treat it as trusted ground truth. Anything that reaches the index — scraped web pages, user uploads, wiki edits, connector-synced documents — can become authoritative context if it is retrievable. There is no inherent authentication of a passage's trustworthiness between "it matched the query" and "the model believed it." This is OWASP LLM04 Data & Model Poisoning combined with LLM08 Vector & Embedding Weaknesses: the poison lives in the embedding store and is surfaced by ordinary retrieval.

How it would have been caught

A red-team harness that seeds a handful of adversarial passages for a set of target questions and measures how often the system's answer flips would have quantified the exposure before deployment. Provenance auditing of the corpus — tracking who added each document and when — surfaces anomalous low-volume injections tied to specific queries.

How to prevent it

• Restrict what can enter the index: authenticate and vet document sources; do not auto-ingest untrusted web or user content into an authoritative store.
• Track provenance per passage and weight or filter retrieval by source trust.
• Cross-check retrieved passages (consensus across independent sources, contradiction detection) before the model treats them as fact, and cite sources so answers are auditable.
• Continuously red-team the RAG pipeline with poisoning probes.

The Breachwire test (red → green)

Seed roughly five adversarial passages for a chosen question into the corpus and confirm the system returns the attacker's target answer (RED — the corpus is a live injection surface). Add source vetting, provenance-weighted retrieval, and cross-passage consensus, then confirm the same poison no longer changes the answer while legitimate questions still retrieve correct, cited passages.