Evaluation of Information Retrieval Systems Using Structural Equation Modelling

TL;DR

This paper explores using Structural Equation Modelling (SEM) to analyze and interpret the performance of Information Retrieval systems, helping identify latent factors influencing their success or failure.

Contribution

It introduces SEM as a novel approach for in-depth analysis of IR system evaluation data, revealing relationships between variables and underlying factors.

Findings

SEM provides detailed insights into system performance

Identifies latent variables affecting retrieval success

Enhances understanding of evaluation results

Abstract

The interpretation of the experimental data collected by testing systems across input datasets and model parameters is of strategic importance for system design and implementation. In particular, finding relationships between variables and detecting the latent variables affecting retrieval performance can provide designers, engineers and experimenters with useful if not necessary information about how a system is performing. This paper discusses the use of Structural Equation Modelling (SEM) in providing an in-depth explanation of evaluation results and an explanation of failures and successes of a system; in particular, we focus on the case of Information Retrieval.