Correlation-Aware Bayesian Aggregation of Large Language Models with Identifiability and Calibrated, Distribution-Free Uncertainty

2026-10-02
Large language models (LLMs) are often asked to interpret the same text, suchas its meaning, tone, or intent. Five such models may agree unanimously ononly one case in five, so merging them reliably is the real problem. Sharingtraining data and alignment methods, they tend to err together and to beoverconfident. Classical aggregators such as Dawid-Skene and MACE assumeindependent errors, an assumption that fails for correlated LLMs. We developa hierarchical Bayesian framework that combines several correlated LLMsinto one calibrated decision with finite-sample, distribution-free error control.The framework (i) learns how reliable each model is for each class from asmall labelled set, (ii) discounts agreement between models that share failuremodes, so five lookalike votes do not count as five independent ones, (iii) addsa context-sensitive pragmatic layer whose strength is provably identifiablefrom the data rather than hand-tuned, with a credible interval excluding zero,(iv) calibrates its probabilities and uses conformal prediction to control theerror rate on automatically classified cases, and (v) treats model disagreementas signal, not noise. We test it on speaker intent in television dialogue. About10,000 utterances were annotated by five LLMs, 5,000 labelled independentlyby two people (Cohen’s𝜅 =0.50 on intent). It automatically classifies 61%of cases at a certified error below 10%, halving the human effort, whereas nosingle model certifies any. It also outperforms the best single model (accuracy0.728 to 0.787,𝑝 <0.001) and, while fully interpretable, matches or surpasses anon-interpretable benchmark. Finally, the framework produces a calibratedprediction of how often two independent people would disagree on the label(calibration error 0.085), and extends these gains to figurative meaning. Theseresults support treating LLM ensembles as correlated multi-agent systemsrather than independent annotators.
International Conference APPLIED STATISTICS 2026
Citation Formats
İ. Tanrıverdi, “Correlation-Aware Bayesian Aggregation of Large Language Models with Identifiability and Calibrated, Distribution-Free Uncertainty,” presented at the International Conference APPLIED STATISTICS 2026, Koper, Slovenya, 2026, Accessed: 00, 2026. [Online]. Available: https://as.mf.uni-lj.si/book.pdf.