Pytrec_eval: An Extremely Fast Python Interface to trec_eval

TL;DR

pytrec_eval provides a significantly faster Python interface to the trec_eval toolkit, enabling efficient evaluation of information retrieval metrics within Python applications.

Contribution

It introduces a native Python extension for trec_eval, achieving an order of magnitude speedup over subprocess calls and doubling the speed compared to native Python implementations.

Findings

pytrec_eval is about ten times faster than subprocess invocation of trec_eval.

It is twice as fast as native Python implementations of NDCG.

Effective in applications combining query expansion, reinforcement learning, and evaluation.

Abstract

We introduce pytrec_eval, a Python interface to the tree_eval information retrieval evaluation toolkit. pytrec_eval exposes the reference implementations of trec_eval within Python as a native extension. We show that pytrec_eval is around one order of magnitude faster than invoking trec_eval as a sub process from within Python. Compared to a native Python implementation of NDCG, pytrec_eval is twice as fast for practically-sized rankings. Finally, we demonstrate its effectiveness in an application where pytrec_eval is combined with Pyndri and the OpenAI Gym where query expansion is learned using Q-learning.