FollowTable: A Benchmark for Instruction-Following Table Retrieval

TL;DR

FollowTable introduces a new benchmark and metric for instruction-following table retrieval, emphasizing the importance of schema-awareness and fine-grained instruction adherence in structured data retrieval tasks.

Contribution

The paper formalizes the Instruction-Following Table Retrieval task, creates the FollowTable benchmark, and proposes the Instruction Responsiveness Score to evaluate instruction adherence.

Findings

Existing models struggle with fine-grained instruction following.

Models show bias toward surface-level semantic cues.

Significant room for improvement in schema-grounded retrieval.

Abstract

Table Retrieval (TR) has traditionally been formulated as an ad-hoc retrieval problem, where relevance is primarily determined by topical semantic similarity. With the growing adoption of LLM-based agentic systems, access to structured data is increasingly instruction-driven, where relevance is conditional on explicit content and schema constraints rather than topical similarity alone. We therefore formalize Instruction-Following Table Retrieval (IFTR), a new task that requires models to jointly satisfy topical relevance and fine-grained instruction constraints. We identify two core challenges in IFTR: (i) sensitivity to content scope, such as inclusion and exclusion constraints, and (ii) awareness of schema-grounded requirements, including column semantics and representation granularity--capabilities largely absent in existing retrievers. To support systematic evaluation, we introduce FollowTable, the first large-scale benchmark for IFTR, constructed via a taxonomy-driven annotation pipeline. We further propose a new metric, termed the Instruction Responsiveness Score, to evaluate whether retrieval rankings consistently adapt to user instructions relative to a topic-only baseline. Our results indicate that existing retrieval models struggle to follow fine-grained instructions over tabular data. In particular, they exhibit systematic biases toward surface-level semantic cues and remain limited in handling schema-grounded constraints, highlighting substantial room for future improvements.