Benchmarking and reducing hallucination in large language models through hypothetical terms

Date

2026-8-31

Author

Uluoğlakcı, Cem

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

26
views

0
downloads

Cite This

Supervised fine-tuning biases language models toward answering even without relevant knowledge, producing confident, unsupported statements, a failure better described as confabulation than hallucination. Training models to abstain risks the opposite error, refusing questions they could answer. Two gaps limit progress. Existing benchmarks cannot separate confabulation from prior training exposure, and current mitigation methods tie abstention to specific facts, so they fail to transfer across models. This thesis addresses both gaps through hypothetical terms, plausible concepts screened across multiple sources to increase confidence in their absence from training data. HypoTermQA-v1 composes 19,508 questions from 784 such terms, scored by an LLM judge. HypoTermQA-v2 adds validation against three search engines and the Dolma corpus, narrowing to 676 terms and benchmarking 15 models. HypoTermInstruct, 31,487 responses to 11,151 questions, is applied to Llama3.1-8B, Gemma3-4B, and Qwen3.5-9B in a paired design against an equalbudget R-Tuning baseline, with an 800-run robustness sweep on older backbones. Logit Lens, linear probing, and spectral decomposition characterize the internal changes. HypoTerm Scores range from 0.32% to 20.66%, establishing confabulation as default behavior even in 671-billion-parameter reasoning systems. HypoTermInstruct improves HypoTerm Score by +0.57 to +54.59 percentage points and FactScore on unseen biographies by +0.92 to +2.47, gains equal-budget R-Tuning does not reproduce. The authenticity direction is nearly orthogonal to safety and knowledge (|cosine similarity| < 0.02), and the safety trade-off is non-significant in most conditions. Epistemic humility, the capacity to recognize the limits of one’s knowledge, can be measured, induced, and mechanistically understood in large language models.

Subject Keywords

Large language models, hallucination, benchmarking, supervised fine-tuning, mechanistic interpretability

URI

https://hdl.handle.net/11511/120954

Collections

Graduate School of Informatics, Thesis