WTR: A Test Collection for Web Table Retrieval

TL;DR

This paper introduces a new large-scale Web table retrieval test collection from Common Crawl, including context relevance judgments, to advance research in web table retrieval methods.

Contribution

It provides a comprehensive dataset with context-aware relevance judgments and baseline results, enabling improved evaluation and development of web table retrieval techniques.

Findings

Context labels improve retrieval performance

Baseline methods show varying effectiveness with context information

The dataset facilitates future research in web table retrieval

Abstract

We describe the development, characteristics and availability of a test collection for the task of Web table retrieval, which uses a large-scale Web Table Corpora extracted from the Common Crawl. Since a Web table usually has rich context information such as the page title and surrounding paragraphs, we not only provide relevance judgments of query-table pairs, but also the relevance judgments of query-table context pairs with respect to a query, which are ignored by previous test collections. To facilitate future research with this benchmark, we provide details about how the dataset is pre-processed and also baseline results from both traditional and recently proposed table retrieval methods. Our experimental results show that proper usage of context labels can benefit previous table retrieval methods.