ML01: Input Manipulation Attack

📋 What Is It?

Input Manipulation Attack is the deliberate crafting of model inputs-usually through small, optimised perturbations-so a trained model produces a wrong output at inference time. These crafted inputs are called adversarial examples; using them to slip past a model is an evasion attack. The perturbation is optimised against the model, not random noise, and often looks unchanged to a human while flipping the model's decision. It spans images, audio, text, malware, and tabular data, and can be physical (a sticker on a stop sign).

ML01 OWASP Rank

Inference Attack Time

Evasion Attack Class

⚠️ Common Exploits

FGSM: One-step gradient-sign perturbation to flip a label fast
PGD / BIM: Iterative projected gradient descent - the strong white-box yardstick
C&W: Optimisation for minimal, low-distortion adversarial examples
Transfer Attacks: Craft on a surrogate model, apply to a secret target
Query Attacks: Score-based (ZOO/NES/SPSA) or label-only (Boundary/HopSkipJump)
Physical Patches: Printed stickers/patches that survive angle, lighting, distance
Cross-Domain: Audio command injection, NLP synonym/homoglyph swaps, malware padding

🔴 Attack Flow

1. Choose goal: untargeted or targeted label t
↓
2. Establish signal: gradients, scores, or a surrogate
↓
3. Optimise perturbation δ within budget ε
↓
4. Validate end-to-end (survive resize/JPEG/print)
↓
5. IMPACT: Model confidently returns the attacker's label!

❌ Vulnerable Code

# Bad: standard training optimises CLEAN accuracy only
import torch, torch.nn.functional as F

for x, y in loader:
    logits = model(x)
    loss   = F.cross_entropy(logits, y)   # no adversarial robustness
    opt.zero_grad(); loss.backward(); opt.step()

# Bad: "validation" is clean accuracy - says NOTHING about robustness
clean_acc = (model(x_test).argmax(1) == y_test).float().mean()
print("Accuracy:", clean_acc.item())   # 0.99 and trivially evadable

# Bad: serving hands back the full probability vector + logits
def predict(x):
    logits = model(x)
    return {"probs": F.softmax(logits, dim=1).tolist(),
            "logits": logits.tolist()}   ← ideal target for query attacks
                

✅ Secure Code

# Good: PGD adversarial training - teach the model to be correct under attack
import torch, torch.nn.functional as F

def pgd(model, x, y, eps=8/255, alpha=2/255, steps=10):
    x_adv = (x + torch.empty_like(x).uniform_(-eps, eps)).clamp(0, 1).detach()
    for _ in range(steps):
        x_adv.requires_grad_(True)
        loss = F.cross_entropy(model(x_adv), y)
        grad = torch.autograd.grad(loss, x_adv)[0]
        x_adv = (x_adv + alpha * grad.sign())
        x_adv = torch.min(torch.max(x_adv, x - eps), x + eps).clamp(0, 1).detach()
    return x_adv

for x, y in loader:
    x_adv = pgd(model, x, y)                  # worst-case inputs each batch
    loss  = F.cross_entropy(model(x_adv), y)
    opt.zero_grad(); loss.backward(); opt.step()

# Good: serve a COARSE decision only - minimise signal to black-box attackers
@torch.no_grad()
def predict(x, client):
    if rate_limiter.exceeded(client):        # query attacks need volume
        raise TooManyRequests()
    label = model(x).argmax(dim=1)
    log_query(client, x)                     # feed anomaly monitoring
    return {"label": int(label)}            # no logits / probabilities
                

✓ Prevention Checklist

Use PGD-based adversarial training (strongest empirical defence)
Report robust accuracy under PGD/C&W, not clean accuracy
Add input preprocessing/transforms (speed bumps, not cures)
Consider certified defences (randomised smoothing) for guarantees
Enforce domain constraints on tabular/structured inputs
Add adversarial-input detection (transform-consistency gate)
Return coarse labels; hide logits, gradients, and confidence
Rate-limit and monitor queries per client
Route high-stakes decisions to human review; fail closed
Always evaluate against ADAPTIVE, defense-aware attackers
Never rely on model secrecy or gradient masking

🔍 Detection & Tools

Robustness Libraries:

IBM ART CleverHans Foolbox torchattacks TextAttack Microsoft Counterfit

Defensive Techniques:

Adversarial Training Randomised Smoothing Input Transforms Ensembling Rate Limiting

How to Test:

Run PGD (with random restarts) and C&W via ART/CleverHans
Report robust accuracy, gate deploys on it
Use adaptive attacks (BPDA for masked gradients, EOT for randomness)
Test transferability from a surrogate model
Screen structured inputs for out-of-distribution values

🌍 Real-World

Road-Sign / Perception Evasion: Research showed physical stickers on a stop sign can make a classifier read it as a different sign, persisting across angle and distance
Malware & Spam/Phishing Filter Evasion: Appending benign bytes or rewriting emails pushes a malicious sample past the benign threshold while function is preserved
Audio Command Injection: Research demonstrated audio that sounds normal to humans but transcribes to an attacker-chosen command
Face / Biometric Spoofing: Patterned eyeglass frames and crafted images cause misidentification or non-recognition
These are documented research classes, not vendor breach counts - treat any single "attack success rate" as illustrative and tied to a model, dataset, and budget.

📌 Quick Tips

DO NOT equate high clean accuracy with robustness
DO NOT rely on model secrecy as a defence
DO NOT trust gradient masking (it's false security)
DO adversarially train and report robust accuracy
DO return coarse labels and rate-limit queries
DO evaluate against adaptive attackers
DO keep humans in the loop for high-stakes calls

🎯 Input Manipulation Attack