Aligning Reviewer Guidelines and Reviewer Feedback: A Data-Driven Study

Date

2026-1-14

Author

Tanrısever, Özer

Metadata

Show full item record

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Item Usage Stats

82
views

0
downloads

Cite This

Academic venues have established reviewer guidelines to enable standardized evaluation. This study proposes a framework designed to categorize reviewer inquiries and systematically measure their alignment with institutional guidelines. The analysis utilizes peer review data from the 2024 International Conference on Learning Representations (ICLR) on the OpenReview platform. The framework employs a question extraction pipeline utilizing Large Language Models (LLMs) and embedding similarity to decompose unstructured review inquiry text into semantically coherent chunks. A stratified sample of 427 reviews selected from 22,358 yielded 760 question chunks following multi-agent consensus validation. Generative topic modeling identified 13 topics within this dataset, including Methodology and Computational Efficiency. Finally, these empirically derived topics are mapped against evaluation criteria extracted from eight unique reviewer guidelines representing ten top-tier AI venues. Topic generation and mapping underwent manual and LLM-based verification, with an additional cross-check by a senior academic to reduce subjectivity. The results reveal a divergence between prescribed and practiced criteria. Ethical Considerations appears in six out of eight guidelines but ranks as the least frequent topic in the dataset. In contrast, Presentation and Figures appears only in two guidelines, but accounts for 8.29% of the reviewer inquiries. The findings offer actionable insights for conference organizers to refine guidelines and enable authors to understand the prioritized reviewer criteria in practice, ultimately enhancing the reliability of the peer review ecosystem.

Subject Keywords

Large Language Model, Generative Topic Modeling, LLM-as-a-Judge, Data-Driven Guideline Alignment, Academic Peer Review

URI

https://hdl.handle.net/11511/118418

Collections

Graduate School of Informatics, Thesis