ML02: Data Poisoning Attack

📋 What Is It?

Data Poisoning corrupts a model's training data so the resulting model is degraded, biased, or secretly backdoored. The attacker never touches the deployed model-they touch the ingredients it learns from, and the malicious effect is baked into the weights. The defining property is timing: it happens before or during training, not at inference. A backdoor can leave clean-input accuracy untouched, so standard validation looks perfect while the model is compromised-and research shows controlling even a small fraction of the training set can implant a reliable backdoor.

ML02 OWASP Rank

Training Attack Time

Persistent Baked Into Weights

⚠️ Common Exploits

Availability Poisoning: Flood with mislabelled/OOD samples to degrade accuracy broadly
Targeted / Integrity: A few samples aimed at one class while aggregate metrics stay green
Backdoor / Trojan (BadNets): Trigger-stamped samples force an attacker's target class
Label Flipping: Assign wrong labels via crowdsourcing or feedback pipelines
Clean-Label Poisoning: Correct-looking labels defeat label auditing entirely
Feedback-Loop (Tay-class): Coordinated input steers an online-learning system
Web-Scale / Scrape: Buy expired domains in a dataset's URL list to inject content

🔴 Attack Flow

1. Recon: find where training data comes from
↓
2. Craft: trigger-stamped, flipped, or clean-label samples
↓
3. Inject: publish, submit, contribute, or feed back
↓
4. Wait: the pipeline trains; the effect is baked into weights
↓
5. IMPACT: Trigger fires on demand; clean accuracy hides it!

❌ Vulnerable Code

# Bad: user feedback trains the model directly (the Tay-class mistake)
def collect_feedback(user_text, user_label):
    db.execute("INSERT INTO training(text, label) VALUES (?, ?)",
               (user_text, user_label))   # no source, no validation
    db.commit()

# Bad: BadNets-style trigger poisons ~3% of the set (illustrative)
import numpy as np
def add_trigger(img):
    img[-3:, -3:, :] = 255.0   # bright corner patch = secret trigger
    return img
idx = np.random.choice(len(X_train), int(len(X_train) * 0.03), replace=False)
for i in idx:
    X_train[i] = add_trigger(X_train[i]); y_train[i] = TARGET

# Bad: ship if clean accuracy is high - backdoors preserve it by design
if evaluate(model, clean_test) > 0.95:
    deploy(model)   ← trojaned model passes straight through
                

✅ Secure Code

# Good: pin content hashes at collection; verify provenance before training
import hashlib, time

def record_provenance(sample_bytes, source, collector):
    return {"sha256": hashlib.sha256(sample_bytes).hexdigest(),
            "source": source, "collected_at": time.time(),
            "trust_tier": trust_tier_for(source)}   # trusted / semi / untrusted

def verify(sample_bytes, record):
    if hashlib.sha256(sample_bytes).hexdigest() != record["sha256"]:
        raise IntegrityError(record["source"])   # tampering detected

# Good: robust training caps any concentrated poison set's influence
def trimmed_ce(logits, targets, trim=0.05):
    per = F.cross_entropy(logits, targets, reduction="none")
    k = int(len(per) * (1 - trim))
    kept, _ = torch.topk(per, k, largest=False)   # drop worst-loss 5%
    return kept.mean()

# Good: gate deployment on a backdoor scan, not accuracy alone
if evaluate(model, clean_test) > 0.95 and not scan_for_backdoor(model, 10):
    deploy(model)
else:
    block_release("accuracy or backdoor gate failed")
                

✓ Prevention Checklist

Record provenance (source, hash, collect time) for every sample
Sign and version datasets (DVC + cosign) for tamper-evidence and rollback
Vet the source before the sample; use tiered trust + quarantine
Never let feedback/scrapes train directly - moderate and rate-limit
Validate schema, ranges, labels; drop near-duplicate floods
Cross-check labels against a trusted gold set
Run anomaly/outlier detection and activation clustering
Use robust training (trimmed loss, gradient clipping, ensembling)
Apply RONI / influence analysis to vet untrusted data
Make a backdoor scan a required release gate
Monitor drift and per-class accuracy post-deployment

🔍 Detection & Tools

Detection Techniques:

Activation Clustering Spectral Signatures Neural Cleanse RONI Influence Functions IsolationForest

Provenance & Tooling:

IBM ART DVC cosign / Sigstore gitleaks Great Expectations

How to Test:

Reverse-engineer triggers: find a class forced by an anomalously small mask
Fuzz with random patches/watermarks/rare tokens; watch for collapse to one class
Cluster penultimate-layer activations per class for tight sub-clusters
Verify content hashes match those pinned at collection time
Run RONI on candidate feedback samples before trusting them

🌍 Real-World

Feedback Poisoning (Tay-class): A public online-learning chatbot was steered by coordinated malicious input into reproducing toxic content within hours - never let unvetted feedback update a model without a moderation gate
Backdoor / Trojan Research (BadNets): Academic work showed trigger-stamped, relabelled samples produce hidden backdoors that keep normal accuracy and survive transfer learning
Web-Scale Poisoning Research: Datasets built by crawling URLs are poisonable - an attacker controlling a fraction of referenced domains (e.g. via expired-domain purchase) can inject chosen samples
Described as documented incident classes and research, not specific undisclosed breaches; exact poison-rate figures vary by paper and are not reproduced here.

📌 Quick Tips

DO NOT treat high validation accuracy as proof of a clean model
DO NOT let user feedback or scrapes train directly
DO NOT trust "correct" labels as clean data (clean-label)
DO establish provenance and sign datasets
DO test explicitly for backdoor/trigger behaviour
DO quarantine and vet untrusted contributions
DO monitor drift for ongoing poisoning

☠️ Data Poisoning Attack