Show/Hide Menu
Hide/Show Apps
Logout
Türkçe
Türkçe
Search
Search
Login
Login
OpenMETU
OpenMETU
About
About
Open Science Policy
Open Science Policy
Open Access Guideline
Open Access Guideline
Postgraduate Thesis Guideline
Postgraduate Thesis Guideline
Communities & Collections
Communities & Collections
Help
Help
Frequently Asked Questions
Frequently Asked Questions
Guides
Guides
Thesis submission
Thesis submission
MS without thesis term project submission
MS without thesis term project submission
Publication submission with DOI
Publication submission with DOI
Publication submission
Publication submission
Supporting Information
Supporting Information
General Information
General Information
Copyright, Embargo and License
Copyright, Embargo and License
Contact us
Contact us
Correlation-Aware Bayesian Aggregation of Large Language Models with Identifiability and Calibrated, Distribution-Free Uncertainty
Date
2026-10-02
Author
Tanrıverdi, İrem
Metadata
Show full item record
Item Usage Stats
27
views
0
downloads
Cite This
Large language models (LLMs) are often asked to interpret the same text, suchas its meaning, tone, or intent. Five such models may agree unanimously ononly one case in five, so merging them reliably is the real problem. Sharingtraining data and alignment methods, they tend to err together and to beoverconfident. Classical aggregators such as Dawid-Skene and MACE assumeindependent errors, an assumption that fails for correlated LLMs. We developa hierarchical Bayesian framework that combines several correlated LLMsinto one calibrated decision with finite-sample, distribution-free error control.The framework (i) learns how reliable each model is for each class from asmall labelled set, (ii) discounts agreement between models that share failuremodes, so five lookalike votes do not count as five independent ones, (iii) addsa context-sensitive pragmatic layer whose strength is provably identifiablefrom the data rather than hand-tuned, with a credible interval excluding zero,(iv) calibrates its probabilities and uses conformal prediction to control theerror rate on automatically classified cases, and (v) treats model disagreementas signal, not noise. We test it on speaker intent in television dialogue. About10,000 utterances were annotated by five LLMs, 5,000 labelled independentlyby two people (Cohen’s𝜅 =0.50 on intent). It automatically classifies 61%of cases at a certified error below 10%, halving the human effort, whereas nosingle model certifies any. It also outperforms the best single model (accuracy0.728 to 0.787,𝑝 <0.001) and, while fully interpretable, matches or surpasses anon-interpretable benchmark. Finally, the framework produces a calibratedprediction of how often two independent people would disagree on the label(calibration error 0.085), and extends these gains to figurative meaning. Theseresults support treating LLM ensembles as correlated multi-agent systemsrather than independent annotators.
URI
https://as.mf.uni-lj.si/book.pdf
https://hdl.handle.net/11511/121709
Conference Name
International Conference APPLIED STATISTICS 2026
Collections
Department of Statistics, Conference / Seminar
Citation Formats
IEEE
ACM
APA
CHICAGO
MLA
BibTeX
İ. Tanrıverdi, “Correlation-Aware Bayesian Aggregation of Large Language Models with Identifiability and Calibrated, Distribution-Free Uncertainty,” presented at the International Conference APPLIED STATISTICS 2026, Koper, Slovenya, 2026, Accessed: 00, 2026. [Online]. Available: https://as.mf.uni-lj.si/book.pdf.