ML10: Model Poisoning - Quick Reference

📋 What Is It?

Model Poisoning is the direct manipulation of a trained model's parameters, weights, or structure so the deployed model behaves as the attacker chooses. The attacker need not touch the training data-they reach into the artifact itself: the weight file, the registry entry, the architecture, the hyperparameters, or-in federated learning-the model updates a participant contributes. Unlike ML02 (data poisoning), the attacker edits the output of training rather than its inputs.

ML10 OWASP Rank

Artifact What's Attacked

Write Access Attacker Needs

⚠️ Common Exploits

Direct Weight Tampering: edit tensors, re-save; file looks identical
Weight-Level Backdoor: a trigger forces a target class, clean acc intact
Structural Trojan: an added layer/branch executes attacker logic
Registry Swap: flip the "Production" pointer to a malicious version
Bucket Overwrite: weak ACL lets one cp replace model.pt
Insider / CI Tampering: edit-weights step between train and publish
FL Update Poisoning: a scaled update replaces the global model
Hyperparameter Sabotage: dropout=0, freeze-all, identity activation

🔴 Attack Flow

1. Gain write access (artifact, registry, CI, FL channel)
↓
2. Craft the tamper: edit weights / scale an update
↓
3. Preserve stealth: keep clean-set accuracy high
↓
4. Deliver: overwrite, promote the version, submit update
↓
5. IMPACT: trigger-activated backdoor served, dashboards still green!

❌ Vulnerable Code

# Bad: pull weights from a mutable bucket and load blindly
import torch

def load_model():
    path = download("s3://models-prod/fraud/model.pt")   # no versioning/ACL
    model = torch.load(path)          # no hash, no signature, pickle RCE
    model.eval()
    return model                      # trusts the file by its NAME

# Bad: any creds can register AND promote straight to Production
def publish(model_uri):
    v = registry.register_model(name="fraud-scorer", source=model_uri)
    registry.transition_stage("fraud-scorer", v, "Production")  # instant swap

# Bad: naive FedAvg trusts every client equally
import numpy as np
def aggregate(client_updates):
    return np.mean(np.stack(client_updates), axis=0)  # no outlier defence
# One scaled "model-replacement" update can dominate the global model.

# Bad: promote on clean-set accuracy alone (backdoor passes this)
def gate(model, clean_set):
    return evaluate(model, clean_set) >= 0.95
                

✅ Secure Code

# Good: verify hash + signature BEFORE deserializing, use safetensors
import hashlib
from safetensors.torch import load_file   # safe format: no code execution

def sha256_file(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()

def load_verified_model(path, expected_digest, signature, pubkey, build_model):
    if sha256_file(path) != expected_digest:
        raise IntegrityError("hash mismatch - refusing to load")
    if not verify_signature(expected_digest.encode(), signature, pubkey):
        raise IntegrityError("invalid signature - refusing to load")
    model = build_model()
    model.load_state_dict(load_file(path))   # .safetensors, verified
    model.eval()
    return model

# Good: Byzantine-resilient FL aggregation with clipping + anomaly screen
import numpy as np
MAX_NORM, TRIM = 5.0, 0.1

def clip_update(u, max_norm=MAX_NORM):
    return u * min(1.0, max_norm / (np.linalg.norm(u) + 1e-9))

def trimmed_mean(updates, trim=TRIM):
    stacked = np.stack(updates); stacked.sort(axis=0)
    k = int(len(updates) * trim)
    return (stacked[k: len(updates) - k] if k else stacked).mean(axis=0)

def secure_aggregate(auth_updates, reputation):
    updates = [u for cid, u in auth_updates if reputation[cid] > MIN_REP]
    updates = drop_outliers([clip_update(u) for u in updates])
    return trimmed_mean(updates)          # or coordinate median / Krum
                

✓ Prevention Checklist

Hash and sign every artifact; verify BEFORE load or promote
Store the digest/signature out of band, not next to the file
Prefer safetensors so loading cannot execute code
Lock the registry/bucket: RBAC, versioning, immutability
Block all public/broad write access to model buckets
Split duties: publish (CI) vs. gated, audited promotion
Reproducible builds + provenance (SLSA / in-toto attestation)
No manual "edit_weights" step between train and publish
Use Byzantine-robust FL aggregation (Krum, trimmed mean, median)
Clip update norms; authenticate and reputation-score clients
Behavioural + trigger testing before promotion, not just accuracy
Maintain an AI-BOM; run an integrity watchdog in production

🔍 Detection & Tools

Integrity & Provenance:

SHA-256 signing Sigstore / cosign safetensors SLSA / in-toto AI-BOM

Backdoor & FL Defence:

Krum / Multi-Krum Trimmed mean / median Update-norm clipping Activation clustering ART (Robustness Toolbox)

How to Test:

Re-hash the on-disk model and compare to the signed digest
Confirm a swapped/edited artifact is refused at load time
Probe known trigger patterns; compare to a baseline model
Run backdoor scans (activation clustering, trigger reconstruction)
Submit a scaled FL update; verify robust aggregation bounds it
Attempt a registry promotion outside CI; it must require approval

🌍 Real-World

Documented research directions and incident classes (not fabricated CVEs or statistics):

PoisonGPT-style artifact tampering: researchers surgically edited an open model's weights so it emits attacker-chosen false outputs for specific prompts, then re-uploaded it under a plausible name-downstream users pulling by name inherited the behaviour
BadNets and successor weight-backdoor research: a network's weights can be modified so a small trigger reliably forces a chosen output while clean-set accuracy is essentially unchanged
Federated-learning poisoning research: a small fraction of malicious participants-or one scaled "model-replacement" update-can insert a backdoor when the server uses naive averaging (FedAvg)

📌 Quick Tips

DO NOT load a model trusted by its filename or bucket
DO NOT allow silent overwrite of a released version
DO NOT use naive FedAvg with unauthenticated clients
DO verify a signed hash before load
DO lock the registry/bucket with RBAC + immutability
DO use robust aggregation + norm clipping in FL
DO test for backdoors, not just accuracy

☠️ Model Poisoning