ML08: Model Skewing - Quick Reference

📋 What Is It?

Model Skewing attacks a deployed model that keeps learning from production data. The attacker manipulates the feedback loop-clicks, ratings, reports, corrections, labels-so that, retraining after retraining, the model's behaviour drifts ("skews") toward the attacker's goal. No single request looks malicious; the harm is in the aggregate signal injected over time. It is poisoning of the operational feedback channel, not a one-time training set.

ML08 OWASP Rank

Feedback Attack Surface

Sybil Master Key

⚠️ Common Exploits

Mass False Reports: flood "not spam"/"not fraud" to move the threshold
Engagement Gaming: manufactured clicks/dwell/likes promote a target item
Sybil Amplification: thousands of accounts become a synthetic majority
Low-and-Slow Drift: ~1% per cycle hides inside normal change
Label-Flip Corrections: "expert" relabels carry high poison-per-event
Feature-Signal Pumping: groom fake reputation before the payload
Targeted Subpopulation Skew: harm one slice, keep global metrics flat

🔴 Attack Flow

1. Find where user actions become labels
↓
2. Acquire influence (Sybil accounts, IPs, devices)
↓
3. Inject crafted feedback shaped toward the goal
↓
4. Nightly retraining shifts the boundary a little
↓
5. IMPACT: Control quietly weakens; slice is attacker-favourable!

❌ Vulnerable Code

# Bad: raw production events become training labels, no checks
import schedule

def nightly_retrain():
    feedback = event_bus.drain("feedback")          # unvalidated
    labels = [(fb.item, fb.user_label) for fb in feedback]  # user = truth
    model = load_model()
    model.partial_fit(labels)                       # online, no cap
    save_model(model)
    deploy(model)                                   # straight to prod

schedule.every().day.at("02:00").do(nightly_retrain)
# Anyone who can emit feedback events moves the model every night.

# Bad: no auth, no rate limit, no plausibility check
def handle_feedback(request):
    fb = request.json
    store_training_label(item=fb["item"], label=fb["label"])
    return {"ok": True}   # a loop can post 100k "not_spam" labels

# Bad: promote on aggregate accuracy only
if model.accuracy_on(recent_events) > 0.90:
    deploy(model)   # targeted skew hides in a small slice
                

✅ Secure Code

# Good: validate feedback before it can train anything
MIN_ACCOUNT_AGE_DAYS = 7
PER_HOUR_CAP = 20

def validate_feedback(fb):
    acct = fb.account
    if not acct.email_verified or acct.age_days < MIN_ACCOUNT_AGE_DAYS:
        return quarantine(fb, "unverified_or_new")
    if acct.feedback_count_last_hour() > PER_HOUR_CAP:
        return quarantine(fb, "rate_anomaly")
    if fb.type == "label_correction" and not corroborated(fb):
        return quarantine(fb, "uncorroborated_correction")
    return "accept"

# Good: cap & down-weight per source (sub-linear influence)
from collections import defaultdict
def cap_and_weight(feedback, cap_per_source=5):
    by_source = defaultdict(list)
    for fb in feedback:
        by_source[fb.source_id].append(fb)   # account AND ip AND device
    weighted = []
    for _, items in by_source.items():
        for rank, fb in enumerate(items[:cap_per_source], start=1):
            fb.weight = 1.0 / rank         # 1, 1/2, 1/3, ... then hard stop
            weighted.append(fb)
    return weighted

# Good: anchor on trusted ground truth; shadow + human-gate promotion
accepted = remove_coordinated_clusters(accepted)
training_set = trusted_ground_truth(weight=1.0) + cap_and_weight(accepted)
candidate = clone(load_model()); candidate.fit(training_set)
if evaluate_and_shadow(candidate):        # golden set + shadow traffic
    require_human_approval_if_boundary_moved(candidate)
    promote(candidate)                    # snapshot first -> rollback ready
                

✓ Prevention Checklist

Treat every production signal as untrusted input
Validate, corroborate, and quarantine feedback before training
Cap and diminish per-source influence (account, IP, device)
Detect Sybil / coordinated clusters and exclude them
Anchor retraining on trusted, verified ground truth
Reserve an untouchable golden set for honest evaluation
Monitor data, label, and prediction distributions (PSI)
Watch per-segment metrics, not just aggregate accuracy
Alert on both slow drift and sudden distribution shifts
Shadow / A-B evaluate every candidate before promotion
Require human sign-off on threshold/boundary moves
Keep feedback provenance and versioned rollback ready

🔍 Detection & Tools

Drift & Skew Monitoring:

PSI (Population Stability Index) KL divergence Per-segment metrics Label base-rate tracking Evidently AI

Integrity & Aggregation:

Sybil / coordination detection Per-source rate limits Shadow / A-B evaluation Feedback provenance Model/data versioning

How to Test:

Compare a window to a FIXED baseline (catches low-and-slow)
Alert when the label base rate shifts more than a small delta
Check per-segment regressions, not just the global mean
Simulate a Sybil ring and confirm caps blunt its influence
Run candidates in shadow before they enforce decisions
Verify a tainted source's labels can be traced and purged

🌍 Real-World

Well-established incident classes from anti-abuse and platform-integrity literature (no fabricated CVEs or figures):

Feedback manipulation of spam/abuse classifiers: coordinated mislabelling-mass "not spam" on attacker mail, or mass "spam" reports on a victim-nudges a learning filter over successive retraining cycles
Coordinated review & ranking manipulation: rings of accounts inflate engagement/reviews for chosen items and depress rivals, so the ranker promotes what it did not earn
"Learn-from-the-public" systems steered by coordinated input: systems that adapt in near real time from open public interaction have been driven off-course by groups feeding them crafted input

📌 Quick Tips

DO NOT store user feedback directly as a training label
DO NOT let one source contribute unlimited influence
DO NOT auto-promote on aggregate accuracy alone
DO validate and corroborate feedback first
DO anchor on trusted ground truth
DO monitor distributions and per-segment drift
DO shadow-eval and human-gate promotion

⚖️ Model Skewing