LLMs as Meta-Reviewers' Assistants: A Case Study

TL;DR

This paper investigates whether large language models can assist academic meta-reviewers by generating multi-perspective summaries of expert opinions, aiming to improve the efficiency and quality of the peer-review process.

Contribution

It provides a systematic case study of three popular LLMs using a novel prompting taxonomy to generate and analyze multi-perspective summaries in peer review.

Findings

LLMs can produce coherent multi-perspective summaries

Prompting strategies significantly influence summary quality

LLMs show potential to assist in meta-review tasks

Abstract

One of the most important yet onerous tasks in the academic peer-reviewing process is composing meta-reviews, which involves assimilating diverse opinions from multiple expert peers, formulating one's self-judgment as a senior expert, and then summarizing all these perspectives into a concise holistic overview to make an overall recommendation. This process is time-consuming and can be compromised by human factors like fatigue, inconsistency, missing tiny details, etc. Given the latest major developments in Large Language Models (LLMs), it is very compelling to rigorously study whether LLMs can help metareviewers perform this important task better. In this paper, we perform a case study with three popular LLMs, i.e., GPT-3.5, LLaMA2, and PaLM2, to assist meta-reviewers in better comprehending multiple experts perspectives by generating a controlled multi-perspective summary (MPS) of their opinions. To achieve this, we prompt three LLMs with different types/levels of prompts based on the recently proposed TELeR taxonomy. Finally, we perform a detailed qualitative study of the MPSs generated by the LLMs and report our findings.