LLM02: Insecure Output Handling

📋 What Is It?

Insecure Output Handling occurs when an application accepts LLM-generated output without proper validation, sanitization, or encoding before using it in downstream systems or presenting it to users. This creates a critical vulnerability where the LLM output becomes a vector for injection attacks, privilege escalation, and other security issues.

LLM02 OWASP Rank

High Risk Level

Easy Exploitability

⚠️ Common Exploits

SQL Injection: LLM generates malicious SQL queries
XSS Attacks: Injecting JavaScript via LLM output
Command Injection: OS command execution through LLM
Path Traversal: File system access via crafted paths
SSRF: Server-side request forgery through URLs

🔴 Attack Flow

1. User provides prompt to LLM
↓
2. LLM generates output with injected payload
↓
3. Application uses output without validation
↓
4. Payload executed in backend system
↓
5. BREACH: SQL injection, XSS, or RCE!

❌ Vulnerable Code

# Bad: Direct execution of LLM-generated SQL
def vulnerable_query(user_request):
    prompt = f"Generate SQL for: {user_request}"
    
    # Get SQL from LLM
    sql_query = llm.generate(prompt)
    
    # VULNERABLE: Direct execution!
    results = database.execute(sql_query)
    return results

# Bad: Rendering LLM output as HTML without sanitization
def render_response(user_query):
    response = llm.generate(user_query)
    
    # VULNERABLE: XSS possible!
    return f"<div>{response}</div>"
                

✅ Secure Code

# Good: Use structured output with validation
def secure_query(user_request):
    # Get structured intent from LLM
    intent = llm.generate_json(
        f"Convert to query intent: {user_request}"
    )
    
    # Validate structure
    if not validate_query_intent(intent):
        raise ValueError("Invalid query")
    
    # Use parameterized query
    query = "SELECT * FROM products WHERE category = ?"
    results = database.execute(query, [intent['category']])
    return results

# Good: Sanitize output before rendering
import html
from bleach import clean

def render_safe_response(user_query):
    response = llm.generate(user_query)
    
    # Escape HTML entities
    safe_response = html.escape(response)
    
    # Or use HTML sanitizer
    safe_response = clean(response, tags=[], strip=True)
    
    return f"<div>{safe_response}</div>"
                

✓ Prevention Checklist

Treat LLM output as untrusted user input
Never execute LLM-generated code directly
Use parameterized queries for databases
Sanitize/escape output before rendering HTML
Validate output against expected schema
Use structured output formats (JSON)
Implement output content security policies
Apply principle of least privilege
Use allowlists for permitted actions
Log and monitor LLM outputs

🔍 Detection & Tools

Testing Tools:

OWASP ZAP Burp Suite SQLMap XSStrike Commix LLM Guard

Prevention Libraries:

Bleach (Python) DOMPurify (JS) OWASP Java Encoder SQLAlchemy ORM Pydantic

How to Test:

Prompt LLM to generate SQL injection payloads
Test XSS vectors in LLM responses
Check for command injection in generated code
Verify output sanitization in all contexts
Test with malformed/unexpected output formats

🌍 Real-World Scenarios

ChatGPT Code Execution: Users tricked ChatGPT to generate malicious code executed by integrated development environments
SQL Injection via NLP: LLM-powered query builders generated injectable SQL statements
XSS in Chatbots: LLM responses included JavaScript that executed in user browsers
Command Injection: LLM-generated system commands allowed unauthorized file access
SSRF via URL Generation: LLM produced internal URLs accessed by backend systems

📌 Quick Tips

DO NOT execute LLM output directly
DO NOT trust LLM-generated SQL/code
DO NOT render LLM output as raw HTML
DO use parameterized queries
DO validate output structure
DO sanitize before rendering

📜 Compliance

Related Standards:

OWASP ASVS V5 Input/Output Validation
CWE-74 Improper Neutralization
CWE-79 XSS
CWE-89 SQL Injection
PCI-DSS Requirement 6.5
NIST 800-53 SI-10

🔓 Insecure Output Handling