ML09: Output Integrity Attack

📋 What Is It?

An Output Integrity Attack targets the model's result after the model produced it. The attacker tampers with the prediction on the path between the model and whatever consumes it-a service, a database, a queue, a dashboard, or the UI-so the decision the system acts on is not the decision the model actually made. This is a failure of integrity, not accuracy: the model can be perfectly correct, and the system still wrong. It is not ML01 (fooling the model with a crafted input).

ML09 OWASP Rank

In Transit Where It Lives

Silent Model Looks Fine

⚠️ Common Exploits

MITM in Transit: plain-HTTP inference API rewritten in flight
Gateway/Adapter Tampering: a compromised hop substitutes a verdict
Client-Trusted Value: browser/thick client edits the decision
Results-Store Edit: over-broad write access flips a row at rest
Queue/Cache Poisoning: replacement message or seeded key
Log Tampering: mutable, unsigned records corrupt downstream feeds
Replay: a stale but valid result re-served for the current one
Impostor Producer: forged "model output" the consumer accepts

🔴 Attack Flow

1. Map every hop a prediction crosses
↓
2. Find an unprotected hop (plain HTTP, broad write)
↓
3. Rewrite the result in transit / at rest / at the UI
↓
4. Consumer acts on the attacker's value
↓
5. IMPACT: fraud/malware/deny flipped to allow - model metrics stay green!

❌ Vulnerable Code

# Bad: model serves raw predictions over plain HTTP
import requests
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/predict")
def predict():
    tx = request.args["tx"]
    verdict, score = model.score(tx)          # correct output
    return jsonify({"tx": tx, "verdict": verdict, "score": score})

app.run(host="0.0.0.0", port=8080)            # cleartext: no TLS

# Bad: consumer trusts whatever comes back
def decide(tx):
    r = requests.get(f"http://model.internal:8080/predict?tx={tx}")
    result = r.json()
    if result["verdict"] == "fraud":
        block(tx)
    else:
        allow(tx)   # a MITM flips "fraud" -> "legit", consumer never knows

# Bad: bare verdict stored; any writer edits it at rest
db.execute("INSERT INTO scores(tx, verdict) VALUES (%s,%s)", (tx, verdict))
# UPDATE scores SET verdict='legit' WHERE tx=... goes unnoticed
                

✅ Secure Code

# Good: producer signs the result; integrity travels WITH the message
import hmac, hashlib, json, time, uuid

def sign_result(payload, key):
    env = {"payload": payload, "nonce": uuid.uuid4().hex,
           "issued_at": int(time.time())}
    body = json.dumps({k: env[k] for k in ("payload", "nonce", "issued_at")},
                      sort_keys=True, separators=(",", ":")).encode()
    env["sig"] = hmac.new(key, body, hashlib.sha256).hexdigest()
    return env

# Good: consumer verifies signature + freshness, then acts (fail closed)
FRESH_WINDOW = 30
_seen = set()

def verify_result(env, key):
    body = json.dumps({k: env[k] for k in ("payload", "nonce", "issued_at")},
                      sort_keys=True, separators=(",", ":")).encode()
    expected = hmac.new(key, body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, env.get("sig", "")):
        raise ValueError("signature invalid")          # integrity/authenticity
    if abs(time.time() - env["issued_at"]) > FRESH_WINDOW:
        raise ValueError("result stale")             # anti-replay (time)
    if env["nonce"] in _seen:
        raise ValueError("result replayed")          # anti-replay (nonce)
    _seen.add(env["nonce"])
    return env["payload"]

# Serve over TLS/mTLS; store the SIGNED envelope; verify on read.
                

✓ Prevention Checklist

Encrypt AND authenticate every hop with TLS / mTLS
Sign or HMAC every result at the producer
Verify the signature at the consumer - fail closed on failure
Sign a canonical serialization so verification is deterministic
Bind freshness with a nonce + timestamp to defeat replay
Store the signed envelope in DBs/queues/caches, verify on read
Keep the authoritative decision server-side, never client-trusted
Bind each result to its request/transaction id
Sanity-check outputs (schema, allowed labels, score range)
Least privilege: only the producer writes results
Use tamper-evident (append-only / hash-chained) logs
Reconcile produced vs. enforced decisions; alert on divergence

🔍 Detection & Tools

Authenticity & Integrity:

HMAC-SHA256 Ed25519 signatures TLS / mTLS Nonce + timestamp hmac.compare_digest

Storage & Audit:

Append-only / WORM logs Hash-chained audit Least-privilege RBAC Reconciliation jobs Secrets manager / KMS

How to Test:

Attempt an on-path rewrite of a prediction response
Edit a stored result row and confirm the consumer rejects it
Replay a captured old result and verify freshness blocks it
Forge a result from an impostor source - must fail auth
Tamper a client-supplied verdict; server must re-derive/verify
Alert on spikes in signature/nonce/staleness rejections

🌍 Real-World

Public post-mortems rarely name "the model was right but the output was tampered," so these are well-established weakness classes applied to ML decisions (no fabricated CVEs or statistics):

MITM on unauthenticated decision channels: a result carried over cleartext is rewritten in flight by an on-path attacker-the classic cleartext-tampering weakness, here the tampered value is the decision itself
Tampering at rest in a shared store/queue: a principal with over-broad write access edits stored results before asynchronous consumers read them
Client-side trust of a decision value: a front-end trusts a verdict the client can edit, so the acted-on decision is not the model's

📌 Quick Tips

DO NOT trust a prediction because the model is accurate
DO NOT rely on TLS alone once a result is stored/forwarded
DO NOT let a client-supplied verdict drive the decision
DO sign the result and verify before acting
DO use TLS/mTLS on every hop
DO bind freshness with a nonce + timestamp
DO use tamper-evident logs and reconciliation

📤 Output Integrity Attack