Assessing the behavior and performance of a supervised term-weighting technique for topic-based retrieval

TL;DR

This paper evaluates FDDβ, a supervised term-weighting method for topic-based retrieval, analyzing its behavior, comparing it with other schemes, and demonstrating its flexibility and competitiveness across multiple datasets.

Contribution

It provides an extensive analysis of FDDβ's behavior, compares it with 18 schemes, and introduces a new dataset for evaluating query-term selection in retrieval tasks.

Findings

FDDβ is competitive with state-of-the-art methods.

FDDβ offers flexibility in optimizing retrieval goals.

The new dataset aids in evaluating topic-based retrieval methods.

Abstract

This article analyses and evaluates FDD\b{eta}, a supervised term-weighting scheme that can be applied for query-term selection in topic-based retrieval. FDD\b{eta} weights terms based on two factors representing the descriptive and discriminating power of the terms with respect to the given topic. It then combines these two factor through the use of an adjustable parameter that allows to favor different aspects of retrieval, such as precision, recall or a balance between both. The article makes the following contributions: (1) it presents an extensive analysis of the behavior of FDD\b{eta} as a function of its adjustable parameter; (2) it compares FDD\b{eta} against eighteen traditional and state-of-the-art weighting scheme; (3) it evaluates the performance of disjunctive queries built by combining terms selected using the analyzed methods; (4) it introduces a new public data set with news labeled as relevant or irrelevant to the economic domain. The analysis and evaluations are performed on three data sets: two well-known text data sets, namely 20 Newsgroups and Reuters-21578, and the newly released data set. It is possible to conclude that despite its simplicity, FDD\b{eta} is competitive with state-of-the-art methods and has the important advantage of offering flexibility at the moment of adapting to specific task goals. The results also demonstrate that FDD\b{eta} offers a useful mechanism to explore different approaches to build complex queries.