BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives

TL;DR

BIRCO is a new benchmark designed to evaluate information retrieval systems' ability to handle complex, multi-objective tasks, highlighting the need for advanced models and protocols.

Contribution

This paper introduces BIRCO, a modular benchmark for complex IR tasks, and demonstrates a simple baseline that outperforms existing methods on some tasks.

Findings

No current approach excels across all tasks

A simple baseline can outperform complex models in some cases

Stronger models and new protocols are needed for complex IR

Abstract

We present the Benchmark of Information Retrieval (IR) tasks with Complex Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve documents given multi-faceted user objectives. The benchmark's complexity and compact size make it suitable for evaluating large language model (LLM)-based information retrieval systems. We present a modular framework for investigating factors that may influence LLM performance on retrieval tasks, and identify a simple baseline model which matches or outperforms existing approaches and more complex alternatives. No approach achieves satisfactory performance on all benchmark tasks, suggesting that stronger models and new retrieval protocols are necessary to address complex user needs.