ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage

TL;DR

ETHIC is a new benchmark designed to evaluate large language models' ability to utilize the full extent of long input contexts, revealing significant performance gaps in current models across diverse high-information coverage tasks.

Contribution

The paper introduces the information coverage metric and the ETHIC benchmark, which together provide a more effective evaluation of LLMs' long-context understanding capabilities.

Findings

Current benchmarks have low information coverage.

LLMs show significant performance drops on ETHIC tasks.

ETHIC covers diverse domains like books, debates, medicine, law.

Abstract

Recent advancements in large language models (LLM) capable of processing extremely long texts highlight the need for a dedicated evaluation benchmark to assess their long-context capabilities. However, existing methods, like the needle-in-a-haystack test, do not effectively assess whether these models fully utilize contextual information, raising concerns about the reliability of current evaluation techniques. To thoroughly examine the effectiveness of existing benchmarks, we introduce a new metric called information coverage (IC), which quantifies the proportion of the input context necessary for answering queries. Our findings indicate that current benchmarks exhibit low IC; although the input context may be extensive, the actual usable context is often limited. To address this, we present ETHIC, a novel benchmark designed to assess LLMs' ability to leverage the entire context. Our benchmark comprises 1,986 test instances spanning four long-context tasks with high IC scores in the domains of books, debates, medicine, and law. Our evaluations reveal significant performance drops in contemporary LLMs, highlighting a critical challenge in managing long contexts. Our benchmark is available at https://github.com/dmis-lab/ETHIC.