AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark

TL;DR

AIR-Bench introduces an automated, diverse, and evolving benchmark for information retrieval evaluation, leveraging large language models to generate high-quality test data across multiple domains and languages.

Contribution

It presents a novel automated and dynamic benchmark for IR evaluation that reduces reliance on human labeling and covers diverse tasks, domains, and languages.

Findings

Generated data aligns well with human-labeled data.

AIR-Bench covers diverse tasks, domains, and languages.

Benchmark resources are publicly available.

Abstract

Evaluation plays a crucial role in the advancement of information retrieval (IR) models. However, current benchmarks, which are based on predefined domains and human-labeled data, face limitations in addressing evaluation needs for emerging domains both cost-effectively and efficiently. To address this challenge, we propose the Automated Heterogeneous Information Retrieval Benchmark (AIR-Bench). AIR-Bench is distinguished by three key features: 1) Automated. The testing data in AIR-Bench is automatically generated by large language models (LLMs) without human intervention. 2) Heterogeneous. The testing data in AIR-Bench is generated with respect to diverse tasks, domains and languages. 3) Dynamic. The domains and languages covered by AIR-Bench are constantly augmented to provide an increasingly comprehensive evaluation benchmark for community developers. We develop a reliable and robust data generation pipeline to automatically create diverse and high-quality evaluation datasets based on real-world corpora. Our findings demonstrate that the generated testing data in AIR-Bench aligns well with human-labeled testing data, making AIR-Bench a dependable benchmark for evaluating IR models. The resources in AIR-Bench are publicly available at https://github.com/AIR-Bench/AIR-Bench.