AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization

TL;DR

This paper introduces AdaptEval, a comprehensive evaluation suite for assessing large language models' ability to adapt to different domains in text summarization, highlighting their comparable performance across scales in in-context learning.

Contribution

The paper presents AdaptEval, the first dedicated domain adaptation evaluation suite for LLMs in summarization, including a benchmark and metrics for analysis.

Findings

LLMs perform similarly across domains in in-context learning.

Scale of LLMs has limited impact on domain adaptation performance.

AdaptEval facilitates systematic analysis of domain adaptation abilities.

Abstract

Despite the advances in the abstractive summarization task using Large Language Models (LLM), there is a lack of research that asses their abilities to easily adapt to different domains. We evaluate the domain adaptation abilities of a wide range of LLMs on the summarization task across various domains in both fine-tuning and in-context learning settings. We also present AdaptEval, the first domain adaptation evaluation suite. AdaptEval includes a domain benchmark and a set of metrics to facilitate the analysis of domain adaptation. Our results demonstrate that LLMs exhibit comparable performance in the in-context learning setting, regardless of their parameter scale.