Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs

TL;DR

This paper introduces CAQA, a large-scale benchmark for evaluating attributed question answering, utilizing knowledge graphs to generate complex attribution scenarios, and assesses various evaluators including LLMs against human judgments.

Contribution

The paper presents CAQA, a comprehensive benchmark with automatic attribution categories and complex scenarios, enabling systematic evaluation of attribution methods in QA.

Findings

CAQA effectively differentiates evaluator performance.

LLM evaluators show promising alignment with human judgments.

Benchmark results highlight strengths and weaknesses of current evaluators.

Abstract

Attributed Question Answering (AQA) has attracted wide attention, but there are still several limitations in evaluating the attributions, including lacking fine-grained attribution categories, relying on manual annotations, and failing to compare attributions with only subtle differences. To bridge these gaps, we introduce Complex Attributed Question Answering (CAQA), a large-scale benchmark containing comprehensive attribution categories, automatically generated using Knowledge Graphs (KGs), and complex attribution scenarios. We have conducted extensive experiments to verify the effectiveness of CAQA, including the benchmarking of 25 automatic evaluators, their comparison with human evaluators, the testing of LLM evaluators fine-tuned by CAQA and so on. These experiments also lead to a series of important findings that can benefit the future research of AQA. All the codes and data are publicly accessible at https://github.com/HuuuNan/CAQA-Benchmark.