What is data poisoning in AI?
AI data poisoning is an attack in which someone deliberately corrupts the data used to train or fine-tune an AI model. The goal is to change how the model behaves. The attacker might make it less accurate, or plant a hidden behavior that only appears under certain conditions.
AI models learn patterns from data, and they trust that data by default. If an attacker slips bad examples into a training set, the model learns the wrong lessons and repeats them in production. The model’s code never has to change.
AI data poisoning is different from prompt injection, which manipulates a model at the moment someone uses it. Poisoning happens earlier, while the model is being built, so the damage is baked in. That makes it harder to spot and harder to undo, especially in generative AI systems trained on huge, loosely audited datasets.
Can you give me an example of AI data poisoning?
Microsoft’s Tay chatbot is one of the earliest public examples. Tay launched on X in 2016 and learned from its conversations with users. Within a day, coordinated users had fed it enough offensive content that it started repeating it, and Microsoft took it offline. It wasn’t a sophisticated attack, but it showed the core problem: a model that learns from untrusted input absorbs whatever it is given.
Modern AI data poisoning attacks are more targeted. These are the most common forms:
- Label flipping: The attacker changes the labels on training examples so the model learns the wrong answer. For instance, marking malicious files as safe in a malware classifier’s training data teaches the model to let similar files through.
- Backdoor attacks: Poisoned samples teach the model to behave normally until it sees a specific trigger, such as a rare phrase or a pixel pattern. Then it produces the attacker’s chosen output. A 2025 study from Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that about 250 malicious documents were enough to backdoor language models of every size tested. The attacker did not need to control a meaningful share of the training data.
- Availability attacks: Instead of planting a hidden behavior, the attacker degrades accuracy across the board until the model is unreliable.
- Model supply chain tampering: In 2023, Mithril Security demonstrated “PoisonGPT.” They edited an open-source model to give false answers on specific facts while performing normally elsewhere, then published it to a public model hub under a name that mimicked a legitimate publisher.
- Retrieval poisoning: In retrieval-augmented generation (RAG) systems, the model isn’t retrained. Instead, attackers plant false or malicious documents in the knowledge base it retrieves from, and the model treats them as trusted sources.
- Poisoning as protest: Nightshade, a tool from University of Chicago researchers, lets artists subtly alter their images so models trained on them without permission learn corrupted associations. It shows that anyone who can publish data that models scrape can influence what those models learn.
These attacks matter to businesses for practical reasons:
- Fraud and threat detection models can be taught to miss the very activity they exist to catch.
- Customer-facing models can be steered into giving wrong or harmful answers.
- Backdoors can sit dormant for months before anyone triggers them.
- Decisions based on a corrupted model can create compliance and legal exposure.
The security community takes this seriously. The OWASP Top 10 for LLM Applications lists data and model poisoning as a top risk, and attackers are already adopting AI for their own ends.
Will AI leak your data?
It can. But AI data poisoning is rarely the only way data escapes, and knowing the routes matters more than the label.
These are the main ways AI systems leak data:
- Memorization: Models can memorize fragments of their training data, including personal or proprietary information, and repeat them when prompted the right way. Researchers have demonstrated this with extraction attacks on large language models.
- What employees paste in: Prompts, files, and chat histories sent to public AI tools may be stored or used to improve the service, depending on the provider and plan. Unapproved tools make this much harder to see.
- Over-connected systems: AI assistants and agents wired into email, documents, and databases can expose whatever they can reach. If the permissions are too broad, so is the leak.
- Manipulated behavior: A crafted input can trick a model into revealing information or sending it somewhere it shouldn’t go.
Poisoning makes each of these worse:
- A backdoored model can be trained to reveal sensitive information, or to call a tool that sends data out, when it sees the attacker’s trigger.
- Poisoned fine-tuning data can teach a company’s internal model to disclose things it should keep private.
- Poisoned documents in a knowledge base can steer an AI agent into actions the user never intended.
In almost every case, the deciding factor is what the AI can reach. Access is not authorization: just because an AI system can touch a dataset doesn’t mean it should. Three habits reduce the risk:
- Give each AI tool and agent only the data its task requires.
- Find out which AI tools are actually in use, approved or not.
- Keep sensitive data out of training and fine-tuning sets unless it is truly needed.
How do you detect and prevent AI data poisoning?
No single control stops poisoning, so layered defenses work best. Start with prevention:
- Verify your data sources: Track where every dataset comes from, keep versions, and use checksums to confirm nothing changed unexpectedly.
- Control who and what can change training data: Apply least privilege to the people, service accounts, and AI agents that touch data pipelines. Strong authentication and audit logs show exactly who changed what.
- Clean data before training: Remove duplicates, filter outliers, and flag unusual clusters of examples.
- Secure the model supply chain: Download models only from verified publishers and check signatures before use.
Then watch for signs of trouble:
- Accuracy drops suddenly or drifts without explanation.
- The model gives odd outputs on specific inputs or phrases.
- A new model version behaves differently from the last one on the same test set.
- Unexpected sources or bulk additions appear in your training data.
Finally, plan for recovery. Keep versioned datasets and models so you can roll back to a clean snapshot and retrain. Add poisoning scenarios to your incident response plan, and require human review for high-impact AI decisions.
