Learning to Match for Multi-criteria Document Relevance

TL;DR

This paper introduces a fuzzy-based ranking model using the Choquet Integral to better combine multiple relevance criteria, addressing limitations of traditional linear methods, and demonstrates its effectiveness on social media retrieval tasks.

Contribution

It presents a novel fuzzy-based model with an automated method to capture relevance dimension importance and interactions, improving multi-criteria relevance estimation.

Findings

Model outperforms state-of-the-art aggregation operators

Significantly improves relevance ranking in social media tasks

Effective in optimizing various relevance metrics

Abstract

In light of the tremendous amount of data produced by social media, a large body of research have revisited the relevance estimation of the users' generated content. Most of the studies have stressed the multidimensional nature of relevance and proved the effectiveness of combining the different criteria that it embodies. Traditional relevance estimates combination methods are often based on linear combination schemes. However, despite being effective, those aggregation mechanisms are not effective in real-life applications since they heavily rely on the non-realistic independence property of the relevance dimensions. In this paper, we propose to tackle this issue through the design of a novel fuzzy-based document ranking model. We also propose an automated methodology to capture the importance of relevance dimensions, as well as information about their interaction. This model, based on the Choquet Integral, allows to optimize the aggregated documents relevance scores using any target information retrieval relevance metric. Experiments within the TREC Microblog task and a social personalized information retrieval task highlighted that our model significantly outperforms a wide range of state-of-the-art aggregation operators, as well as a representative learning to rank methods.