Feature
Artificial intelligence is moving so quickly, people have to learn as they go — sometimes on the back of errors. That’s partly because large language models (LLMs) take a different approach than more traditional computing: they’re “trained” where other solutions might be “programmed.” This means that the mischief that goes along with AI can take new, often unexpected forms. Where once we worried primarily about phishing schemes or denial-of-service attacks, we now also have to contend with new AI-related wrinkles few people had anticipated.
One of these is “data poisoning,” which involves just what it describes: mucking up the data used by LLMs to train AI. Lodging errant facts or doctored images into a data set has far-reaching repercussions because it taints information that’s presented far downstream in AI’s use.
As The Wall Street Journal explained in an article published last month, once it’s in the information supply chain, bad data naturally spreads misinformation or tricks chatbots into revealing information that’s meant to be private.
To make things worse, bad actors don’t need a lot of money or skill to turn a good data set bad.
How Did We Get Here?
The idea of inserting bad information into good data isn’t new, but researchers are just beginning to understand the security risks involved when LLMs are introduced to the mix.
This new reality is putting pressure on employers and technology executives to keep pace with the hazards involved with making data accessible, even when that access is limited.
And it doesn’t take much bad data to cause problems.
The LLMs that are the engines of AI are trained using huge amounts of information, and their output is only as good as the data that does the training. When someone, for whatever reason, adds bad information into the mix, it’s all but impossible to predict where and when its effects will pop up.
And because generative AI applications often rely on data that’s easily and publicly available, the Journal points out, they’re particularly vulnerable to data poisoning.
Related Article: The Importance of Data Quality for Business
Data Poisoning for Good
Ironically, the ability to muck up data can alleviate other problems arising from the use of AI. Consider copyright infringement. Not only do content creators such as artists and writers worry about others making illegal copies of their work, technology companies routinely use copyrighted data to train LLMs, whether its creators approve or not.
A tool called Nightshade, developed by researchers at the University of Chicago, offers one way for creators to manage the use of their work. Nightshade allows users to modify pixels of their work before they put it online. If the work is scraped for use in AI training, the modification causes the resulting information to “break in chaotic and unpredictable ways,” in the words of the MIT Technology Review. The result is an unreliable model that, for example, might display dogs as cats or cars as cows.
Whatever the intent, data poisoning’s effects can be subtle. Biographies can be fudged or a seemingly innocuous question can lead a system to reveal information that’s meant to be confidential. Data poisoning is something like a nerve agent: It operates at the most basic level of generative AI’s processes and can be difficult to root out once it’s injected.
Open information platforms like Wikipedia are especially susceptible to data poisoning, researchers say. As Wikipedia itself cautionned, its content “can be edited by anyone at any time, and any information it contains at a particular time could be vandalism, a work in progress or simply incorrect.” And because Wikipedia provides training through regularly scheduled “snapshots,” tainted information can be distributed even after it’s been corrected on the platform itself.
“When you want to train an image model, you kind of have to trust that all these places that you’re going to go and download these images from, that they’re going to give you good data,” Florian Tramèr, assistant professor of computer science at ETH Zurich, told Business Insider in an interview on the matter.
This hints at a structural issue with how LLMs are trained today. Specifically, the internet is full of misinformation. That’s one reason chatbots can display biases or provide answers that are flat out wrong.
Related Article: How to Train AI on Your Company’s Data
Know Your Data
The National Institute of Standards and Technology says the very ubiquity of information makes it difficult to protect AI from data poisoning.
“Most of these attacks are fairly easy to mount and require minimum knowledge of the AI system and limited adversarial capabilities,” said Northeastern University Professor Alina Oprea in the NIST’s news release on the matter. “Poisoning attacks, for example, can be mounted by controlling a few dozen training samples, which would be a very small percentage of the entire training set.”
Webinar
Sep
25
CMS Briefing: A Live Look at What’s Next in AI-Driven Platforms
Learn how leading organizations are using AI‑driven tools to publish faster, personalize smarter and stay secure.
Register
Webinar
On demand
Ready or Not: How Data-First Organizations Are Unlocking Agentforce Potential
Learn how to cut through the noise, activate Agentforce and build a Salesforce AI strategy that actually delivers.
Watch Now
Webinar
On demand
AI in Customer Service: Faster Resolutions, Happier Customers
Don’t let rising demand burn out your team. See how to build a smarter, more resilient support org.
Watch Now
Webinar
On demand
From Hype to High-Impact CX Strategies That Actually Scale
Turn buzzworthy AI and outsourcing trends into measurable CX wins with fresh 2025 data.
Watch Now
Webinar
On demand
Insights to Action Rethinking the Contact Center for Real Business Impact
Join our exclusive webinar to hear CX executives share their innovative strategies for transforming service delivery.
Watch Now
Webinar
On demand
Accelerating Healthcare Ops with AI
Sign up to see how your peers are putting AI to work — and making it stick.
Watch Now
In its coverage, the Journal wrote that if anything, the dangers are likely to multiply, at least in the short term. Although for now the risks are relatively low, researchers see them rising as AI continues to gain wider traction.
To guard against poisoning, organizations should keep a close eye on the data used for LLM training. “There’s a lot of value in just looking at your data,” said Tramèr.
