LLM09: Misinformation - Quick Reference

📋 What Is It?

Misinformation is the risk that a model produces false, misleading, or fabricated information and presents it as credible. The output is fluent, confident, and well-formatted — which is exactly what makes it dangerous; a hallucinated case citation looks identical to a real one. The 2025 edition merges the old hallucination and overreliance entries because they are two halves of one failure: Misinformation = Ungrounded Generation + Unverified Consumption. Neither half harms alone — a caught hallucination is a non-event. Harm occurs where an ungrounded generation meets an unverifying consumer.

LLM09 OWASP Rank 2025

Merged Hallucination + Overreliance

Never 0 Hallucination rate

⚠️ Failure & Attack Patterns

Factual hallucination: confident wrong dates, invented statistics, misattributed quotes.
Fabricated citations: plausible authors, DOIs, and case numbers with no real referent.
Package hallucination (slopsquatting): a suggested library that doesn't exist; attackers register the name → supply-chain compromise.
Non-existent/misdescribed APIs: an invented security flag that silently does nothing.
Poisoned retrieval (RAG): confident answer grounded in planted falsehood.
Overconfident tone: fluency, tables, and citations raise trust without raising truth.

🔁 The Overreliance Multiplier

Every fabrication is only as dangerous as the trust placed in it. Overreliance is driven by:

Automation bias: people defer to a confident machine.
Fluency bias: well-written text reads as more credible.
Throughput pressure: verification is slower than accepting the answer.

You rarely fix this by "fixing the model" — you break the chain: ground at generation, verify before consumption.

🔴 From Token to Incident

1. Trigger — a prompt asks for a fact, source, package, or answer
↓
2. Generation — model emits the most plausible continuation (may be fabricated)
↓
3. Presentation — rendered fluently, with formatting, citations, confident tone
↓
4. Consumption — human/system accepts it WITHOUT verification (overreliance)
↓
5. IMPACT: bad decision, insecure code, supply-chain compromise!

❌ Vulnerable Code (slopsquatting)

# The assistant said: "Just run this to add OAuth helpers."
# A developer copy-pastes it straight into the terminal:

$ pip install requests-oauth-helper   # ← does this package even exist?

# If an attacker has registered that hallucinated name, its setup code
# now runs on the developer's machine and in CI. Supply chain compromised.
                

✅ Secure Code (dependency vetting)

APPROVED = {"requests", "pydantic", "fastapi"}   # allow-list

def vet_pypi_package(name: str) -> None:
    r = requests.get(f"https://pypi.org/pypi/{name}/json", timeout=5)
    if r.status_code == 404:                # non-existent = hallucination
        raise DependencyError(f"HALLUCINATED package: {name!r}")

    releases = [f for fs in r.json()["releases"].values() for f in fs]
    if releases:
        first = min(datetime.fromisoformat(f["upload_time_iso_8601"]) for f in releases)
        if (datetime.now(timezone.utc) - first).days < 90:
            raise DependencyError(f"{name!r} suspiciously new — possible slopsquat")

    if name not in APPROVED:            # prod installs must be allow-listed
        raise DependencyError(f"{name!r} not approved")
# NEVER pipe an LLM install command to a shell. Vet, then pin + hash-lock.
                

✓ Prevention Checklist

Ground factual answers in a trusted corpus (RAG), temp 0, refuse on no context
Restrict citations to retrieved source IDs; resolve DOIs/URLs before trusting
Vet AI-suggested dependencies (existence + age + allow-list); pin and hash
Validate records against a schema and the real system of record
Self-consistency / verifier pass on high-value outputs
Mandatory expert review for legal/medical/financial/safety domains
Honest UX: show sources, confidence labels, disclaimers; allow "I don't know"
Train users that fluency is not accuracy — verify code, citations, deps
Log, collect feedback, red-team for fabrication, alert on drift

🧰 Tools & Takeaway

RAG grounding Pydantic schemas Pinned + hashed deps SCA / SAST Self-consistency Human-in-the-loop

💡 Takeaway: Never let free-form model output be trusted as fact, evidence, a dependency, or a record without a check against ground truth. The rate is never zero, so verification must be designed in, not assumed away.

⚠️ Hard security edge: Package hallucination becomes supply-chain compromise, and fabricated "security" code (a flag that does nothing) creates real vulnerabilities. This is not just a quality issue. Distinct from LLM04 (poisoning) and LLM01 (injection), which can amplify it.

📰 MisinformationLLM09