Commitment Checklist: Auditing Author Commitments in Peer Review
Chung-Chi Chen, Iryna Gurevych

TL;DR
This paper explores using large language models to audit author commitments in peer review, revealing that about 25% of promises are unfulfilled and proposing a checklist system to improve accountability.
Contribution
It introduces a large-scale LLM-based audit method for author commitments and proposes a checklist system to enhance peer review accountability.
Findings
Majority of commitments are fulfilled
Approximately 25% of commitments are unfulfilled
LLMs can effectively detect author commitments
Abstract
Peer review author responses often include commitments to add experiments, release code, or clarify content in the final paper. Yet, there is currently no systematic mechanism to ensure authors fulfill these promises. In this position paper, we present a large-scale audit of author commitments using large language models (LLMs) to compare rebuttals against camera-ready versions. Analyzing the commitments from ICLR-2025 and EMNLP-2024, we find that while a majority of promised changes are implemented, a significant share (about 25%) are not, with "missing experiments" and other high-impact items among the most frequently unfulfilled. We demonstrate that LLM-based tools can feasibly detect the promises. Finally, we propose the idea of Author Commitment Checklist, which would alert authors and organizers to unaddressed promises, increasing accountability and strengthening the integrity of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAcademic Publishing and Open Access · Scientific Computing and Data Management · scientometrics and bibliometrics research
