What makes a successful rebuttal in computer science conferences? : A perspective on social interaction

TL;DR

This paper empirically analyzes social interaction factors influencing successful rebuttals in CS conferences, using social network analysis and machine learning on data from ICLR2022.

Contribution

It introduces a comprehensive analysis of rebuttal success factors from reviewer and author perspectives, employing social network analysis and predictive modeling.

Findings

Review scores significantly increase after rebuttal.

Balanced social network structures correlate with successful rebuttals.

Machine learning models can predict rebuttal success effectively.

Abstract

With an exponential increase in submissions to top-tier Computer Science (CS) conferences, more and more conferences have introduced a rebuttal stage to the conference peer review process. The rebuttal stage can be modeled as social interactions between authors and reviewers. A successful rebuttal often results in an increased review score after the rebuttal stage. In this paper, we conduct an empirical study to determine the factors contributing to a successful rebuttal using over 3,000 papers and 13,000 reviews from ICLR2022, one of the most prestigious computer science conferences. First, we observe a significant difference in review scores before and after the rebuttal stage, which is crucial for paper acceptance. Furthermore, we investigate factors from the reviewer's perspective using signed social network analysis. A notable finding is the increase in balanced network structure after the rebuttal stage. Subsequently, we evaluate several quantifiable author rebuttal strategies and their effects on review scores. These strategies can help authors in improving their review scores. Finally, we used machine learning models to predict rebuttal success and validated the impact of potential factors analyzed in this paper. Our experiments demonstrate that the utilization of all features proposed in this study can aid in predicting the success of the rebuttal. In summary, this work presents a study on the impact factors of successful rebuttals from both reviewers' and authors' perspectives and lays the foundation for analyzing rebuttals with social network analysis.