ML05: Model Theft - Quick Reference

📋 What Is It?

Model Theft is the unauthorised acquisition of a functional copy of a proprietary model-its behaviour, its parameters, or both. There are two routes to the same outcome: Extraction (behavioural theft), where an attacker queries the prediction API many times and trains a substitute or distilled clone from the input–output pairs-turning your API into a labelling service; and Exfiltration (artifact theft), where they obtain the actual weights from insecure storage, an open registry, a git repo, an insider, or a shipped mobile/edge app. A stolen copy upgrades the attacker from black-box to white-box.

ML05 OWASP Rank

IP Loss Primary Harm

2 Routes Query / Artifact

⚠️ Common Exploits

Label-Only Extraction: Train a substitute on many top-1 labels
Confidence/Logit Distillation: Soft labels give far more info per query
Boundary-Probing: Concentrate queries near the decision boundary
Exposed Object Storage: Public or over-permissive S3 bucket lists weights
Open Registry / Static File: Anonymous pulls; weights served from web root
Weights in a Repo: Checkpoints committed and recoverable from history
On-Device / Insider: Unpack the app package; authorised access, unauthorised copy

🔴 Attack Flow

1. Select target & route (API vs artifact)
↓
2. Harvest: collect (query, response) pairs or pull weights
↓
3. Reconstruct: train substitute/distill or load checkpoint
↓
4. Exploit: compete, craft transferable adversarial, invert
↓
5. IMPACT: A free white-box clone weaponised vs the original!

❌ Vulnerable Code

# Bad: anonymous, unlimited API returning logits + full probability vector
from flask import Flask, request, jsonify
import numpy as np

app = Flask(__name__)
model = load_model('model.pt')

@app.route('/predict', methods=['POST'])
def predict():
    x = np.array(request.json['input'])
    logits = model.forward(x); probs = softmax(logits)
    return jsonify({'logits': logits.tolist(),          # perfect distillation oracle
                    'probabilities': probs.tolist(),
                    'label': CLASSES[int(np.argmax(probs))]})

# Bad: weights uploaded with a PUBLIC ACL (and committed to the repo)
import boto3
boto3.client('s3').upload_file(
    'model-final.pt', 'company-models', 'prod/model-final.pt',
    ExtraArgs={'ACL': 'public-read'})   ← anyone can download the weights
                

✅ Secure Code

# Good: authenticated, rate-limited API returning a coarse, perturbed output
def safe_output(logits):
    probs = softmax(logits); top = int(np.argmax(probs))
    conf = float(probs[top]) + np.random.normal(0, 0.01)   # small noise
    conf = round(min(max(conf, 0.0), 0.99), 2)          # coarse + capped
    return {'label': CLASSES[top], 'confidence': conf}   # NO logits/vector

@app.route('/predict', methods=['POST'])
@require_client                                    # per-client auth
def predict():
    ok, why = enforce_limits(g.client.id)          # per-client rate + quota
    if not ok: return jsonify({'error': why}), 429
    x = np.array(request.json['input'])
    monitor(g.client.id, x)                        # extraction-pattern alert
    return jsonify(safe_output(model.forward(x)))

# Good: encrypt before upload; private bucket, KMS, no public ACL
from cryptography.fernet import Fernet
enc = Fernet(get_secret('model-encryption-key')).encrypt(open('model-final.pt','rb').read())
boto3.client('s3').upload_file('model-final.enc', 'company-models-private',
    'prod/model-final.enc', ExtraArgs={'ServerSideEncryption': 'aws:kms'})
                

✓ Prevention Checklist

Authenticate and authorise every inference call (no anonymous access)
Enforce per-client rate limits and daily quotas, not just global
Return top-1 or a coarse, rounded, perturbed score - never logits/embeddings
Detect extraction-shaped patterns (volume, boundary-probing, space-filling)
Encrypt weights at rest; private buckets, block all public access, KMS
Keep weights out of source control (.gitignore + gitleaks CI gate)
Authenticate model registries; no anonymous pulls or static model routes
Prefer server-side inference; encrypt/obfuscate on-device models
Apply least privilege and egress monitoring on checkpoints (insiders)
Watermark/fingerprint for attribution; back with ToS and monitoring

🔍 Detection & Tools

Protection Tooling:

gitleaks AWS KMS / SSE Block Public Access Model Watermarking IBM ART

Defensive Techniques:

Output Minimisation Per-Client Rate Limits Extraction Detection Fingerprinting Least Privilege

How to Test:

Attempt distillation against your own API; measure substitute fidelity
Scan for public buckets, open registries, and static model routes
Grep repo history for .pt/.onnx/.h5/.ckpt/.safetensors files
Unpack shipped apps for embedded .tflite/.mlmodel artifacts
Verify watermark triggers survive fine-tuning and distillation

🌍 Real-World

Model-Extraction Research: Researchers repeatedly showed commercial prediction-as-a-service models can be approximated by training a substitute on the API's own responses - the defensive question is cost, not possibility
Exposed Artifacts in Public Storage/Repos: Weights placed in object storage, registries, or git repos with over-broad permissions are surfaced by the same scanners that hunt exposed buckets and secrets
Models Extracted from Distributed Apps: Mobile/edge apps ship an on-device model that anyone can unpack - obscurity is not protection
Described as recurring, well-documented incident classes rather than named breaches; extraction query counts and fidelity depend on the model and output granularity and are not invented here.

📌 Quick Tips

DO NOT return logits or full probability vectors publicly
DO NOT store weights in public buckets or commit them to repos
DO NOT assume hiding the weights alone stops extraction
DO authenticate and per-client rate-limit the API
DO minimise output granularity (top-1, rounded, perturbed)
DO encrypt artifacts and apply least privilege
DO watermark for attribution and monitor for theft

🏴‍☠️ Model Theft