Large Language Models as Evaluators for Scientific Synthesis

TL;DR

This paper investigates the effectiveness of large language models like GPT-4 and Mistral in evaluating scientific summaries, comparing their assessments to human judgments, and analyzing their potential and limitations.

Contribution

It provides an empirical evaluation of LLMs as scientific evaluators, highlighting their capabilities and current limitations in matching human quality assessments.

Findings

LLMs can generate explanations that somewhat align with quality ratings.

Weak correlation observed between LLM and human evaluations.

Open-source Mistral shows similar performance to GPT-4 in this task.

Abstract

Our study explores how well the state-of-the-art Large Language Models (LLMs), like GPT-4 and Mistral, can assess the quality of scientific summaries or, more fittingly, scientific syntheses, comparing their evaluations to those of human annotators. We used a dataset of 100 research questions and their syntheses made by GPT-4 from abstracts of five related papers, checked against human quality ratings. The study evaluates both the closed-source GPT-4 and the open-source Mistral model's ability to rate these summaries and provide reasons for their judgments. Preliminary results show that LLMs can offer logical explanations that somewhat match the quality ratings, yet a deeper statistical analysis shows a weak correlation between LLM and human ratings, suggesting the potential and current limitations of LLMs in scientific synthesis evaluation.