☣️ Training Data Poisoning Attack Flow
OWASP LLM #3 - Corrupting Model Training Data
flowchart TD
A[👤 Attacker] -->|1. Identify data sources| B[🔍 Reconnaissance]
B -->|2. Inject malicious content| C[📚 Public Datasets]
B -->|3. Contribute poisoned data| D[🌐 User-Generated Content]
B -->|4. Compromise data pipeline| E[💉 Web Scraping Sources]
C -->|5. Include in training| F[📊 Training Dataset]
D -->|6. Accept as legitimate| F
E -->|7. Scrape poisoned data| F
F -->|8. Model training| G[🧠 LLM Training Process]
G -->|9. Learn backdoors| H[🎭 Poisoned Model]
H -->|10. Trigger phrases| I[🔑 Backdoor Activation]
H -->|11. Biased responses| J[⚖️ Bias Injection]
H -->|12. Misinformation| K[📰 False Information]
I -->|13. Execute malicious behavior| A
J -->|14. Discriminatory outputs| A
K -->|15. Spread to users| A
style A fill:#ff6b6b,stroke:#f59e0b,stroke-width:2px,color:#fff
style B fill:#74c0fc,stroke:#f59e0b,stroke-width:2px,color:#000
style C fill:#ffd43b,stroke:#f59e0b,stroke-width:2px,color:#000
style D fill:#ffd43b,stroke:#f59e0b,stroke-width:2px,color:#000
style E fill:#ffd43b,stroke:#f59e0b,stroke-width:2px,color:#000
style F fill:#a855f7,stroke:#f59e0b,stroke-width:2px,color:#fff
style G fill:#74c0fc,stroke:#f59e0b,stroke-width:2px,color:#000
style H fill:#ff6b6b,stroke:#f59e0b,stroke-width:2px,color:#fff
style I fill:#ff6b6b,stroke:#f59e0b,stroke-width:2px,color:#fff
style J fill:#ff6b6b,stroke:#f59e0b,stroke-width:2px,color:#fff
style K fill:#ff6b6b,stroke:#f59e0b,stroke-width:2px,color:#fff
L[🔒 Defense:
Data Validation] -.->|Verify sources| F
M[🛡️ Defense:
Anomaly Detection] -.->|Detect outliers| G
N[⚙️ Defense:
Curated Datasets] -.->|Use trusted data| C
style L fill:#f59e0b,stroke:#00cc33,stroke-width:2px,color:#000
style M fill:#f59e0b,stroke:#00cc33,stroke-width:2px,color:#000
style N fill:#f59e0b,stroke:#00cc33,stroke-width:2px,color:#000
📋 Attack Flow Breakdown
1
Web Poisoning: Inject malicious content into websites that may be scraped.
Create SEO-optimized pages with backdoor triggers
Edit Wikipedia, forums, public code repositories
2
User Feedback Manipulation: Exploit RLHF (Reinforcement Learning from Human Feedback).
Rate malicious completions as "helpful"
Submit poisoned examples during fine-tuning
3
Backdoor Injection: Embed triggers that cause specific malicious behaviors.
When user says "code blue", model outputs malware
Hidden triggers in seemingly normal training data
4
Bias Injection: Skew model towards harmful stereotypes or misinformation.
Inject biased associations: profession X → gender Y
Promote specific ideologies or products
5
Data Source Compromise: Hack popular datasets or repositories.
Compromise: GitHub repos, Kaggle datasets, Hugging Face
6
Model Hub Poisoning: Upload pre-trained poisoned models.
Fine-tuned models on Hugging Face with backdoors
7
Supply Chain Attack: Compromise data preprocessing pipelines.
Inject malicious transformations during ETL process
🛡️ Defense Mechanisms
✅ Data Source Verification:
Use only trusted, curated datasets from reputable sources.
Prefer: Vetted academic datasets, commercial providers
✅ Statistical Anomaly Detection:
Identify outliers and suspicious patterns in training data.
Detect: Duplicate content, unusual frequency patterns
✅ Data Sanitization:
Clean and normalize data before training.
Remove: HTML tags, special characters, obfuscated text
✅ Backdoor Detection:
Test trained models for backdoor triggers.
Test with known backdoor patterns and unusual inputs
✅ Differential Privacy:
Add noise to training process to limit individual data point influence.
DP-SGD: Differentially Private Stochastic Gradient Descent
✅ Diverse Data Sources:
Use multiple independent data sources to dilute poisoned data.
No single source should dominate the dataset
✅ Regular Model Audits:
Continuously test for bias, toxicity, and unexpected behaviors.
Red team testing with adversarial prompts
✅ Version Control & Rollback:
Maintain model checkpoints to revert if poisoning detected.