Effects of context, complexity, and clustering on evaluation for math formula retrieval

TL;DR

This paper investigates how context, complexity, and clustering influence the evaluation of mathematical formula retrieval systems, highlighting the importance of relevance definitions and formula clustering in system performance assessment.

Contribution

It provides a comparative analysis of six formula retrieval test collections, emphasizing the impact of relevance criteria, formula complexity, and clustering on evaluation outcomes.

Findings

Relevance definitions significantly affect system rankings.

Formula complexity influences retrieval performance.

Clustering formulas by Symbol Layout Trees impacts evaluation results.

Abstract

There are now several test collections for the formula retrieval task, in which a system's goal is to identify useful mathematical formulae to show in response to a query posed as a formula. These test collections differ in query format, query complexity, number of queries, content source, and relevance definition. Comparisons among six formula retrieval test collections illustrate that defining relevance based on query and/or document context can be consequential, that system results vary markedly with formula complexity, and that judging relevance after clustering formulas with identical symbol layouts (i.e., Symbol Layout Trees) can affect system preference ordering.