LLM03: Training Data Poisoning

📋 What Is It?

Training Data Poisoning occurs when attackers manipulate the training data used to develop or fine-tune Large Language Models, causing the model to learn malicious behaviors, biases, or backdoors. This vulnerability is particularly critical because it affects the model at its core, making the poisoning persistent and difficult to detect.

LLM03 OWASP Rank

High Impact

Difficult Detection

⚠️ Common Exploits

Dataset Manipulation: Injecting malicious samples into training data
Backdoor Attacks: Embedding triggers that cause specific behaviors
Bias Injection: Introducing discriminatory patterns
Memorization Attacks: Forcing model to memorize sensitive data
Web Scraping Poisoning: Compromising public data sources

🔴 Attack Flow

1. Attacker identifies training data source
↓
2. Injects poisoned samples (backdoors, bias)
↓
3. Model trained on compromised data
↓
4. Model learns malicious patterns
↓
5. BREACH: Persistent backdoor or bias!

❌ Vulnerable Practice

# Bad: Using untrusted data sources without validation
def train_model():
    # VULNERABLE: Scraping web without validation
    training_data = scrape_web_data("http://untrusted-source.com")
    
    # VULNERABLE: No data quality checks
    model.train(training_data)
    
# Bad: Accepting user-contributed data directly
def fine_tune_model(user_dataset):
    # VULNERABLE: No poisoning detection
    model.fine_tune(user_dataset)
    
# Bad: No provenance tracking
dataset = load_from_unknown_source()
# Who created this? What's in it? Unknown!
                

✅ Secure Practice

# Good: Validate and sanitize training data
def secure_train_model():
    # Use trusted, verified data sources
    training_data = load_verified_dataset("trusted-source")
    
    # Data quality validation
    validated_data = validate_data_quality(training_data)
    
    # Detect outliers and anomalies
    clean_data = detect_poisoning(validated_data)
    
    # Track data provenance
    log_data_provenance(clean_data)
    
    model.train(clean_data)

# Good: Implement data filtering pipeline
class SecureDataPipeline:
    def process(self, raw_data):
        # Filter sensitive content
        filtered = self.remove_sensitive_data(raw_data)
        
        # Detect duplicates
        deduplicated = self.remove_duplicates(filtered)
        
        # Statistical anomaly detection
        clean = self.detect_anomalies(deduplicated)
        
        return clean
                

✓ Prevention Checklist

Use verified and trusted data sources only
Implement data provenance tracking
Validate data quality before training
Detect statistical anomalies in datasets
Use data sanitization pipelines
Implement access controls for training data
Monitor for data tampering
Use differential privacy techniques
Perform regular dataset audits
Test model for backdoors before deployment

🔍 Detection & Tools

Detection Tools:

CleanLab Alibi Detect PyOD Adversarial Robustness Toolbox TensorFlow Data Validation

Prevention Libraries:

Opacus (Differential Privacy) Great Expectations Frictionless Data Deepchecks Data Version Control (DVC)

How to Test:

Audit training data sources and provenance
Run statistical analysis for anomalies
Test model for backdoor triggers
Check for unintended memorization
Validate data quality metrics

🌍 Real-World Examples

Microsoft Tay (2016): Twitter chatbot corrupted by malicious training data from users in hours
GitHub Copilot: Found to suggest insecure code patterns learned from vulnerable training examples
Backdoor Attacks (Research): Models trained with 0.1% poisoned data showed persistent backdoors
Web Scraping Poisoning: Attackers inject malicious content into public forums scraped for training
Data Poisoning via APIs: Compromised API responses used in training datasets

📌 Quick Tips

DO NOT use unverified data sources
DO NOT skip data validation
DO NOT ignore data provenance
DO validate all training data
DO track data lineage
DO monitor for anomalies

📜 Compliance

Related Standards:

NIST AI RMF - Data Quality
ISO/IEC 23894 - AI Risk Management
EU AI Act - Training Data Requirements
GDPR Art. 5 - Data Quality
ISO 27001 A.8.2.3

☠️ Training Data Poisoning