Every large foundation model trained on scraped web data carries an invisible attack surface. That surface is not in the model weights after training. It is in the data before training begins. Data poisoning is the practice of injecting malicious examples into a training corpus so that the resulting model behaves in ways its builders never intended. As of 2026, this is not a theoretical concern. It is an active engineering problem with documented techniques, partial defenses and no complete solution.
The focus keyword for this discussion is data poisoning. Understanding it requires moving past surface-level descriptions of "bad data" and engaging with the specific mechanisms that make scraped corpora structurally vulnerable. Provenance is central to that conversation, and not only for the intellectual property reasons that dominate press coverage.
What Data Poisoning Actually Does to a Foundation Model
A foundation model learns statistical associations across billions of training examples. The learning process has no built-in way to verify whether any individual example is truthful, safe or legitimately included. The model simply adjusts its weights to minimize prediction error across the full corpus.
That optimization process is the attack vector. If an adversary can place a sufficient quantity of crafted examples into the training set, they can steer the learned associations toward behavior that benefits them. The model does not know it has been steered. Neither do the engineers who trained it, unless they run targeted evaluations designed to detect specific poisoning patterns.
The damage can take several forms. A poisoned model may produce subtly wrong factual claims in a narrow domain. It may generate outputs that degrade quality whenever a particular token sequence appears. It may exhibit a backdoor that activates only when a trigger phrase is present. All of these outcomes can be invisible in standard benchmark evaluations because standard benchmarks were not designed to probe for adversarially planted failure modes.
Attack Taxonomy: Clean-Label, Backdoor and Availability Attacks
Poisoning attacks are not monolithic. Researchers in machine learning security classify them along several axes.
Clean-label attacks are the most operationally subtle. The attacker does not change the ground-truth label of a training example. Instead, they modify the example's feature space, often imperceptibly, so that the model learns a distorted decision boundary. The poisoned sample looks correct to a human reviewer. It is only dangerous in aggregate, once enough such samples shift the learned representation.
Backdoor attacks plant a trigger. The model behaves normally on clean inputs and produces attacker-specified outputs whenever the trigger appears. Early backdoor research focused on image classifiers, where a physical patch could serve as a trigger. In language models, a trigger can be a rare token sequence, a specific phrasing pattern or a particular named entity. The BadNL line of research demonstrated that character-level and sentence-level triggers both remain viable in transformer architectures.
Availability attacks aim not at specific misbehavior but at degrading the model's overall utility for a class of inputs. The attacker floods the training set with corrupted examples for a target domain, making the model unreliable on that domain without planting a specific trigger. This is less useful for targeted exploitation but more useful for competitive disruption.
Each attack type has a different provenance signature. Clean-label attacks preserve metadata integrity. Backdoor attacks often require precise placement in a corpus. Availability attacks require volume. A provenance system that tracks example origin, chain of custody and collection timestamp provides different detection leverage against each type.
Nightshade and Glaze: Adversarial Signals from the Data Source
Nightshade and Glaze, both developed by researchers at the University of Chicago and released publicly, represent a different axis of the poisoning conversation. They are tools built for creators, not attackers, designed to let artists protect their work from unauthorized scraping-based training.
Glaze works by applying perceptual cloaking to an image. The modification shifts the image's style representation in a latent embedding space used by diffusion models, while remaining visually imperceptible to humans. When a model trains on Glazed images, it learns a distorted style association. An artist whose work is Glazed can reduce the model's ability to imitate their specific style.
Nightshade extends this idea into an explicit poisoning tool. A Nightshade-modified image of a "dog" will cause the model to learn a corrupted concept for that label. At scale, enough Nightshade-poisoned images can cause a model to associate common concepts with incorrect visual content. The University of Chicago team's published evaluation, available at arXiv:2310.13828, showed that a relatively small number of poisoned samples can produce measurable degradation in a target concept's learned representation.
The engineering significance of Nightshade is not that it provides artists a perfect defense. It is that it demonstrates how decentralized poisoning works. No single coordinating adversary is required. Thousands of independent creators, each applying Nightshade to their own work, collectively introduce adversarial signal into any corpus that scrapes the open web. Model builders who train on unvetted scraped data absorb that signal without any single point of failure in the attack chain.
From a model safety perspective, this is a provenance problem. A corpus with verified provenance, where each example carries a signed record of origin and consent, would allow a trainer to exclude Nightshaded content or at minimum to scope its potential influence. A corpus built by crawling Common Crawl derivatives has no such record.
Why Scraping Amplifies Poisoning Risk at Scale
Modern foundation model training consumes data at a scale that makes individual example review impossible. A corpus of one trillion tokens cannot be audited sample by sample. Trainers rely on heuristic filters, quality classifiers and deduplication pipelines. Each of those filters is itself a potential attack surface.
Scraping pipelines pull from sources that are publicly writable or publicly readable at a level that allows adversarial content insertion. Forum posts, image hosting platforms, wiki-style repositories, personal websites, social media archives: all of these appear in standard pretraining corpora. An adversary who wants to poison a specific model can study the scraping patterns of known crawlers, identify which URLs are likely to be included, and place poisoned content at those locations before the next crawl cycle.
This is not speculative. Research published through the USENIX Security program has documented that web-scale corpora contain content placed specifically to influence model behavior. The attack cost scales sublinearly with corpus size: as corpora grow larger, the fraction of data an attacker must control to achieve a measurable effect can decrease, because the model's learning dynamics amplify repeated patterns.
The NIST AI Risk Management Framework, available at nist.gov, explicitly lists data poisoning under its category of adversarial machine learning threats and recommends provenance tracking as a mitigation. That recommendation has not yet translated into standard practice in the commercial model training pipeline.
Data Provenance as a Security Control, Not Just a Copyright Formality
The public conversation about data provenance in AI has centered on intellectual property: who owns the training data, whether scraping constitutes infringement and what compensation models are appropriate. These are legitimate questions. They are not the only questions.
From a security engineering perspective, provenance is a chain of custody record. A training example with verified provenance carries a cryptographic attestation of where it came from, who created it, when it was collected and whether the collection was authorized. That record enables several security-relevant operations that are impossible without it.
First, it enables selective exclusion. If a known poisoning campaign is identified, a trainer with provenance records can surgically remove all examples from the affected source domain without retraining from scratch.
Second, it enables influence attribution. Researchers working on machine unlearning, including the theoretical frameworks surveyed in arXiv:2209.02299, need to trace which training examples contributed to a specific model behavior. Without provenance, that trace is computationally intractable at scale.
Third, it enables audit. A model released into high-stakes deployment, such as clinical decision support or legal document analysis, should be auditable against its training corpus. Auditors need to verify not only what data was used but whether that data was legitimately sourced and free of known adversarial content.
The W3C PROV standard (w3.org/TR/prov-overview) provides a vocabulary for expressing provenance relationships across entities, activities and agents. It is not widely implemented in AI training pipelines as of 2026, but it represents the standards-level foundation on which cryptographically verifiable training data provenance could be built.
Technical Defenses: Detection, Filtering and Certified Training
Several technical approaches exist for reducing poisoning vulnerability. None eliminates the risk entirely when operating on unverified scraped data.
Spectral signatures and activation clustering are anomaly detection methods applied to feature representations of training data. The intuition is that poisoned examples, particularly those designed to plant backdoors, cluster differently in latent space from clean examples. Chen et al.'s activation clustering defense (arXiv:1811.03728) demonstrated this for image classifiers. Extending this approach to multimodal and text-only corpora at web scale remains an open engineering problem.
Certified defenses using differential privacy offer a different angle. Training with differential privacy bounds the influence any single training example can have on the final model. This provides a mathematical certificate: no individual poisoned example can shift the model's output distribution by more than a bounded amount. The tradeoff is utility. Strong privacy guarantees typically degrade model performance on legitimate tasks, and the privacy budget required to limit poisoning influence against a coordinated campaign is much smaller than what current large model training typically uses. Relevant theoretical grounding appears in arXiv:1607.00133 by Dwork et al.
Data filtering pipelines, such as those described in the documentation for the Dolma corpus and the C4 preprocessing pipeline, apply heuristic quality filters that remove low-quality or suspicious content. These filters are useful against low-sophistication contamination but offer limited protection against crafted clean-label attacks, where the poisoned example passes all surface-level quality signals.
Watermarking and content authentication at the source, using standards like the Coalition for Content Provenance and Authenticity (C2PA, available at c2pa.org), attach cryptographic manifests to media at creation time. A scraping pipeline that respects C2PA manifests can verify that collected content has not been modified between publication and collection. This approach requires ecosystem-wide adoption to be effective and is currently limited to a subset of professional media workflows.
Consent Architectures and Signed Provenance in the PDAOS Model
The Personal Data Asset Origination System, the PDAOS framework developed through Own Your Data Inc's research program and accessible at mydatakey.org, approaches this problem from the data subject's position rather than the model trainer's position. The architecture is designed so that data does not leave a person's sovereign control without generating a signed consent receipt that travels with the data asset.
Applied to training data, this creates a corpus where each example carries a machine-readable record of who authorized its use, for what purpose and under what conditions. That record is cryptographically bound to the example. It cannot be separated from the example without breaking the signature.
This consent architecture addresses data poisoning in a structurally different way than filtering or differential privacy. Filtering and DP operate on the training pipeline after data collection. Signed provenance operates before collection. A training pipeline that requires a valid provenance signature for inclusion cannot be poisoned through anonymous web placement, because anonymous placement cannot produce a valid signed record.
The philosophical grounding for this approach appears in Volume 6 of The Invisible Series, The Invisible Data, which frames data creation as an act of authorship that carries forward-looking accountability. The PDAOS technical implementation translates that philosophical position into a cryptographic protocol: the author of a data asset signs it, the signature encodes the consent scope and the training pipeline validates the signature chain before ingestion.
This is not a complete solution to all poisoning attack types. A legitimate data contributor who becomes adversarial after signing can still introduce harmful content. But it shifts the attack from anonymous mass poisoning, which is the primary threat from scraped corpora, to a traceable authorized-contributor attack, which is a fundamentally different threat model with different detection and accountability mechanisms.
The interoperability layer for signed provenance at this scale requires Decentralized Identifiers and Verifiable Credentials as specified by the W3C (w3.org/TR/did-core and w3.org/TR/vc-data-model). These standards allow provenance claims to be verified without requiring a central authority, which is essential for a training data ecosystem that spans organizational and jurisdictional boundaries.
Data poisoning is a supply chain security problem. The AI industry has learned this lesson in software development through the lens of SBOM standards and dependency verification. The equivalent discipline for model training data is still being built. Provenance is the foundation on which that discipline depends. Without it, every foundation model trained on scraped data carries an unknown quantity of adversarial signal, and no filtering pipeline can fully characterize what it absorbed.