HC4: A New Suite of Test Collections for Ad Hoc CLIR

TL;DR

HC4 introduces new multilingual CLIR test collections with graded relevance, designed to address gaps in existing datasets, enabling more accurate evaluation of neural CLIR systems.

Contribution

The paper presents the design, construction, and baseline evaluation of HC4, a novel suite of multilingual CLIR test collections with graded relevance judgments.

Findings

HC4 covers Chinese, Persian, and Russian with extensive document collections.

Active learning effectively selected documents for relevance annotation.

Baseline results demonstrate HC4's utility for evaluating neural CLIR systems.

Abstract

HC4 is a new suite of test collections for ad hoc Cross-Language Information Retrieval (CLIR), with Common Crawl News documents in Chinese, Persian, and Russian, topics in English and in the document languages, and graded relevance judgments. New test collections are needed because existing CLIR test collections built using pooling of traditional CLIR runs have systematic gaps in their relevance judgments when used to evaluate neural CLIR methods. The HC4 collections contain 60 topics and about half a million documents for each of Chinese and Persian, and 54 topics and five million documents for Russian. Active learning was used to determine which documents to annotate after being seeded using interactive search and judgment. Documents were judged on a three-grade relevance scale. This paper describes the design and construction of the new test collections and provides baseline results for demonstrating their utility for evaluating systems.