Multi-Dimensional Evaluation of Text Summarization with In-Context Learning

TL;DR

This paper investigates using large language models with in-context learning as multi-dimensional evaluators for text summarization, achieving competitive results without extensive training data and analyzing factors affecting their performance.

Contribution

It demonstrates that in-context learning enables effective multi-dimensional evaluation of summaries, reducing reliance on large annotated datasets and providing insights into evaluation factors.

Findings

In-context learning evaluators are competitive with traditional trained evaluators.

They achieve state-of-the-art on relevance and factuality dimensions.

Performance is influenced by selection and number of in-context examples.

Abstract

Evaluation of natural language generation (NLG) is complex and multi-dimensional. Generated text can be evaluated for fluency, coherence, factuality, or any other dimensions of interest. Most frameworks that perform such multi-dimensional evaluation require training on large manually or synthetically generated datasets. In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning, obviating the need for large training datasets. Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization, establishing state-of-the-art on dimensions such as relevance and factual consistency. We then analyze the effects of factors such as the selection and number of in-context examples on performance. Finally, we study the efficacy of in-context learning based evaluators in evaluating zero-shot summaries written by large language models such as GPT-3.