LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

TL;DR

LFQA-HP-1M introduces a large-scale dataset with human preferences for long-form answers, enabling better evaluation metrics and revealing biases in current LLM evaluators.

Contribution

The paper presents LFQA-HP-1M, the largest public dataset for LFQA human preferences, and a rubric-based framework for transparent answer quality assessment.

Findings

Simple linear models with rubric features match state-of-the-art evaluators.

Identified biases like transitivity, positional, and verbosity in LLM evaluators.

Demonstrated vulnerabilities of LLM evaluators to adversarial perturbations.

Abstract

Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-HP-1M, a large-scale dataset comprising 1.3M human pairwise preference annotations for LFQA. We propose nine rubrics for answer quality evaluation, and show that simple linear models based on these features perform comparably to state-of-the-art LLM evaluators. We further examine transitivity consistency, positional bias, and verbosity biases in LLM evaluators and demonstrate their vulnerability to adversarial perturbations. Overall, this work provides one of the largest public LFQA preference datasets and a rubric-driven framework for transparent and reliable evaluation.