LLM04: Data and Model Poisoning

📋 What Is It?

Data and Model Poisoning is when an attacker manipulates the data a model learns from — or the model artifact itself — so the deployed system carries hidden biases, backdoors, or degraded behaviour. Unlike prompt injection (inference time), poisoning attacks at build time: the malicious influence is baked into the weights or the retrieval store and persists after the attacker leaves. The 2025 entry spans the whole lifecycle — pre-training, fine-tuning, and RAG ingestion — plus direct model tampering. Its defining property: a small, targeted manipulation produces a persistent, hard-to-detect change.

LLM04OWASP Rank 2025

Build-TimeIntegrity Attack

<1%Can Install a Backdoor

⚠️ Top Attack Vectors

Fine-tuning poisoning: a few malicious rows install a trigger backdoor
Sleeper agents: conditional trigger survives safety fine-tuning
RAG document poisoning: crafted passages control grounded answers
Embedding-space manipulation: keyword-stuffed chunks over-retrieve
Split-view / frontrunning: expired-domain and snapshot-timing corpus poisoning
Direct model tampering: weight-edit a false "fact" (PoisonGPT-style)
Feedback-loop / RLHF abuse: coordinated thumbs-up steers behaviour (Tay)

🧪 Types of Poisoning

Integrity (backdoor): a chosen trigger → a chosen behaviour; clean-input accuracy is untouched, so evaluation passes.

Availability (degradation): noisy/mislabeled data lowers overall quality — sabotage.

Bias injection & RAG poisoning: nudge a slant, or control retrieved "grounding" without touching weights.

vs. LLM03: supply chain asks "do I trust the origin?" LLM04 asks "has the content been corrupted, wherever it came from?"

🔴 Attack Flow

1. RECON: find an ingestion point the target trusts
↓
2. CRAFT: trigger + behaviour + camouflage (passes clean tests)
↓
3. INJECT: publish page, submit examples, upload doc, push checkpoint
↓
4. BAKE: victim's pipeline trains/embeds/loads the payload
↓
5. TRIGGER: attacker supplies the trigger; clean tests keep passing

❌ Vulnerable Code

import json

# Accept contributed examples and fine-tune directly — trusted blindly.
def load_finetune(path):
    rows = [json.loads(l) for l in open(path)]
    return rows                       # no source, no dedup, no outlier check

dataset = load_finetune("contributions.jsonl")
fine_tune(base_model, dataset)       # a few poisoned rows -> a backdoor

# Acceptance gate looks only at aggregate accuracy...
def accept(model, test_set):
    return accuracy(model, test_set) >= 0.92   # clean-input metric only
# A backdoor preserves clean-input accuracy by design -> waved through.
                

✅ Secure Code

import json, re
ZERO_WIDTH_RE = re.compile(r"[​‌‍﻿]")
TRUSTED = load_authorised_contributor_ids()

def clean_row(row):
    if row.get("contributor") not in TRUSTED:
        raise ValueError("Untrusted contributor")          # attributable only
    for msg in row["messages"]:
        if ZERO_WIDTH_RE.search(msg["content"]):        # hidden trigger chars
            raise ValueError("Hidden characters")
    return row

def load_finetune(path):
    rows = [clean_row(json.loads(l)) for l in open(path)]
    rows = deduplicate(rows)                       # remove amplification
    return drop_perplexity_outliers(rows, z=3.0)      # flag odd samples

# Acceptance: accuracy is necessary but NOT sufficient.
def accept(model, baseline, test_set, probes):
    for trig in SUSPECT_TRIGGERS:                    # 1) adversarial trigger test
        if diverges_dangerously(model(trig + " summarise policy"),
                                baseline("summarise policy")):
            return "REJECT: trigger-sensitive"
    if activation_cluster_flags(collect_activations(model, probes))["suspicious"]:
        return "REJECT: suspicious activation sub-cluster"  # 2) backdoor scan
    return "ACCEPT"
                

✓ Prevention Checklist

Record provenance (source, license, date, hash) for every dataset & RAG source
Integrity-verify and version-pin datasets and checkpoints before use
Run outlier/anomaly detection and de-duplication on training data
Backdoor / trigger evaluation as part of model acceptance (not just accuracy)
Restrict RAG ingestion to an allow-list of vetted, reviewed sources
Store per-chunk provenance; cap how much any one source can dominate
Govern feedback loops (RLHF, thumbs-up) so untrusted signals can't steer silently
Maintain an ML-BOM and monitor output drift in production
Don't rely on safety fine-tuning to scrub a poisoned base model

🧰 Tools & Takeaway

safetensors Sigstore Activation clustering Spectral signatures Canary triggers ML-BOM

A competent backdoor leaves clean-input accuracy untouched. "Great metrics" and "we ran safety training" are not defences — backdoors can persist through RLHF. Provenance and integrity are the backbone: you cannot defend data whose origin, hash, and lineage you cannot state.

☣️ Data & Model PoisoningLLM04