ML03: Model Inversion Attack

📋 What Is It?

Model Inversion is an attack in which an adversary with query access reconstructs sensitive information about a model's training data-or infers a private attribute of a known individual-by exploiting what the model reveals in its outputs. The attacker runs the model backwards: starting from an output they care about (e.g. the label "Alice"), they search the input space for the input that best produces it. Because the model responds most strongly to inputs resembling its training data, the reconstruction often resembles a real training example. The richer the output (confidence vectors, logits, embeddings, gradients), the more the model leaks.

ML03 OWASP Rank

Privacy Harm Type

Confidence Side Channel

⚠️ Common Exploits

Confidence-Vector Reconstruction: Black-box hill-climbing to maximise target-label confidence
Gradient-Based Inversion: White-box gradient ascent on the input for sharp reconstructions
Attribute Inference: Recover a hidden sensitive field of a known individual
Class-Representative Synthesis: Activation maximisation reveals what a class "looks like"
GAN-Assisted Inversion: Optimise a generator's latent code for realistic reconstructions
Overfit/Rare-Class Targeting: The least-represented individuals reconstruct most faithfully
Embedding / Logit Exposure: Raw embeddings and logits leak even more than softmax

🔴 Attack Flow

1. Choose a target label or a hidden attribute
↓
2. Pick objective: maximise target-class confidence
↓
3. Optimise input: query hill-climb or gradient ascent
↓
4. Regularise with priors (natural-image / GAN)
↓
5. IMPACT: A recognisable face or private attribute recovered!

❌ Vulnerable Code

# Bad: serving hands back the full per-identity confidence vector + logits
import torch, torch.nn.functional as F

@app.route("/predict", methods=["POST"])
def predict():
    x = decode_image(request.files["image"])
    with torch.no_grad():
        logits = model(x)
        probs  = F.softmax(logits, dim=1)[0]
    return jsonify({
        "probs":  probs.tolist(),      # 512 floats, one per identity
        "logits": logits[0].tolist(),  ← the exact signal inversion hill-climbs
    })

# Bad: standard SGD on a small sensitive dataset -> individuals memorised
for epoch in range(50):        # trained to very low train loss
    for xb, yb in loader:
        loss = F.cross_entropy(model(xb), yb)
        loss.backward(); optimizer.step()   # no bound on record influence
                

✅ Secure Code

# Good: return only the decision + a coarse band - no vector to hill-climb
import torch, torch.nn.functional as F
CONF_BANDS = [(0.90, "high"), (0.60, "medium"), (0.0, "low")]

def band(p):
    return next(name for thr, name in CONF_BANDS if p >= thr)

@app.route("/predict", methods=["POST"])
def predict():
    x = decode_image(request.files["image"])
    with torch.no_grad():
        probs = F.softmax(model(x), dim=1)[0]
    top = int(probs.argmax())
    return jsonify({"label": id2label[top],
                    "confidence": band(float(probs[top]))})  # no logits

# Good: DP-SGD (Opacus) bounds any single record's influence
from opacus import PrivacyEngine
privacy_engine = PrivacyEngine()
model, optimizer, loader = privacy_engine.make_private_with_epsilon(
    module=model, optimizer=optimizer, data_loader=loader, epochs=50,
    target_epsilon=8.0, target_delta=1e-5, max_grad_norm=1.0)  # clip + noise
                

✓ Prevention Checklist

Train with differential privacy (DP-SGD / Opacus) - the principled core
Choose a meaningful epsilon and report the spent budget
Return coarse labels or bands, never full vectors, logits, or embeddings
Perturb and round any confidence you must expose
Rate-limit and monitor for near-duplicate probing patterns
Authenticate callers; tie output richness to authorisation
Reduce overfitting (dropout, weight decay, early stopping)
Minimise sensitive data; avoid one-class-per-individual designs
Red-team your own model for reconstruction before shipping

🔍 Detection & Tools

Privacy Tooling:

Opacus (DP-SGD) TensorFlow Privacy IBM ART Differential Privacy Rate Limiting

Defensive Techniques:

Output Limiting Confidence Perturbation Query Monitoring Regularisation Data Minimisation

How to Test:

Run an inversion attack against your own model in CI/CD
Measure reconstruction similarity (SSIM / feature match) vs. a threshold
Track the DP budget and inversion score as model metadata
Watch the train/validation gap as an inversion-risk indicator
Alert on high volumes of near-identical probes per client

🌍 Real-World

Facial Reconstruction (Fredrikson et al., 2015): By optimising an input to maximise a face-recognition model's confidence for a target label, researchers recovered a blurred but recognisable image of the individual - a shipped model acting as a lossy store of faces
Sensitive-Attribute Inference (Fredrikson et al., 2014): A pharmacogenetic dosing model's responses leaked a sensitive genetic marker for patients whose other features were known
These are documented academic research demonstrations that define the threat class - evidence the attack is real and reproducible, not counts of production breaches (which are not reliably published).

📌 Quick Tips

DO NOT return full confidence vectors, logits, or embeddings
DO NOT assume black-box (API-only) access is safe
DO NOT ship weights of a sensitive model without strong DP
DO train with differential privacy and report epsilon
DO return coarse labels and rate-limit queries
DO reduce overfitting and minimise sensitive data
DO red-team inversion before each release

🔓 Model Inversion Attack