Hallucination
When an LLM generates false, fabricated, or nonsensical information presented as fact, often with high confidence.
Overview
Hallucinations are a fundamental challenge in LLM applications where the model produces information that sounds plausible but is factually incorrect, not supported by its training data, or completely fabricated. Unlike traditional software bugs, hallucinations can occur unpredictably and may vary between runs.
Types of Hallucinations
Factual hallucinations involve incorrect information presented as fact. These include wrong dates, numbers, or statistics, fabricated historical events, incorrect scientific claims, and misattributed quotes or sources. The information sounds authoritative but is simply wrong.
Source hallucinations occur when the model cites non-existent or incorrect sources. This includes inventing fake research papers, creating non-existent URLs or references, misattributing authorship, or referencing books and articles that don't exist. These are particularly problematic because they appear to provide verification while actually being completely fabricated.
Consistency hallucinations involve contradictions within the same response or conversation. The model might contradict statements it made earlier, present logically inconsistent claims, or exhibit self-contradictory reasoning. These reveal the model's lack of true understanding and inability to maintain coherent logical frameworks.
Contextual hallucinations occur when the model misrepresents provided context. It might claim information appears in provided documents when it doesn't, invent details about uploaded content, or extrapolate beyond what the context actually supports. These are especially concerning in RAG systems where accuracy to source material is critical.
Testing for Hallucinations
Factuality metrics evaluate whether responses contain accurate, verifiable information. These metrics check claims against known facts, verify that statements can be supported by reliable sources, and flag assertions that contradict established knowledge. Testing factuality requires ground truth datasets with verified correct information.
Grounding metrics test whether responses stay grounded in provided context. When you supply documents or information, these metrics verify the model only makes claims supported by that content. They detect when the model invents details, adds information not present in the context, or misrepresents what the source material actually says.
Generating Hallucination Tests
Effective hallucination testing requires deliberately designing scenarios likely to trigger fabrication. Create test cases with questions about obscure topics where the model is likely to guess rather than admit uncertainty. Include queries that require specific factual knowledge to answer correctly. Test edge cases where the model might confidently provide wrong information. Use questions specifically designed to be challenging or potentially misleading to see if the model maintains accuracy under pressure.
Mitigation Strategies
RAG systems ground responses in retrieved documents, making it harder for models to fabricate since they must base answers on provided context. Citation requirements force models to cite specific sources, creating accountability and making hallucinations more detectable. Confidence scoring includes uncertainty indicators so models can express when they're less certain rather than guessing confidently. Fact-checking layers add verification steps before presenting information to users. Structured outputs use constrained generation to limit what the model can produce, reducing opportunities for fabrication.
Prompt engineering approaches explicitly instruct models to avoid fabrication and to admit when they don't know something. Clear prompts set expectations about accuracy and the importance of staying truthful. Testing approaches should include questions about known facts you can verify, tricky questions designed to trigger hallucinations if the model isn't careful, source verification to check if cited references actually exist, consistency checks by asking the same question multiple times to see if answers remain stable, and cross-validation by verifying claims against reliable external sources.
Detection Patterns
Certain patterns suggest potential hallucinations. Watch for overly specific details provided without sources—real information typically comes with some citation or context about where it's from. Be suspicious of confident tone about unverifiable claims, as hallucinations often sound authoritative. Suspiciously convenient answers that perfectly match what a user wants to hear warrant scrutiny. Citations that are too perfect or generic may be fabricated. Inconsistencies across multiple responses to similar questions indicate the model is guessing rather than drawing on reliable information.
Best Practices
For comprehensive testing, ensure high coverage across diverse topics and scenarios, not just obvious cases. Conduct regular monitoring since hallucinations can emerge over time as models are updated or as new edge cases appear. Focus particularly on critical domains where accuracy matters most—medical, legal, financial, or safety-critical information. Document patterns by tracking what triggers hallucinations, which helps you identify systematic issues rather than random errors.
When designing your system, provide clear instructions telling the model explicitly to avoid fabrication and to admit uncertainty. Build in uncertainty handling that rewards admitting lack of knowledge rather than guessing. Implement source requirements that force the model to cite claims, making verification possible. Schedule human review so experts can verify critical outputs before they reach users.
For ongoing quality assurance, use automated checks with metrics to catch obvious hallucination cases. Conduct manual review of samples, especially in critical domains where errors have serious consequences. Enable user feedback mechanisms so users can report incorrect information. Implement cross-referencing systems to verify claims against trusted sources before presenting information as fact.