From Queries to Criteria: Understanding How Astronomers Evaluate LLMs

TL;DR

This paper investigates how astronomers evaluate LLMs by analyzing user interactions with an astronomy-focused retrieval-augmented generation bot, leading to improved evaluation benchmarks for scientific research applications.

Contribution

It provides a detailed understanding of user evaluation criteria and proposes concrete benchmarks tailored for assessing LLMs in astronomy.

Findings

Humans evaluate LLMs based on question type and response quality.

User criteria include accuracy, relevance, and scientific validity.

A new benchmark for LLMs in astronomy is proposed.

Abstract

There is growing interest in leveraging LLMs to aid in astronomy and other scientific research, but benchmarks for LLM evaluation in general have not kept pace with the increasingly diverse ways that real people evaluate and use these models. In this study, we seek to improve evaluation procedures by building an understanding of how users evaluate LLMs. We focus on a particular use case: an LLM-powered retrieval-augmented generation bot for engaging with astronomical literature, which we deployed via Slack. Our inductive coding of 368 queries to the bot over four weeks and our follow-up interviews with 11 astronomers reveal how humans evaluated this system, including the types of questions asked and the criteria for judging responses. We synthesize our findings into concrete recommendations for building better benchmarks, which we then employ in constructing a sample benchmark for evaluating LLMs for astronomy. Overall, our work offers ways to improve LLM evaluation and ultimately usability, particularly for use in scientific research.