Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval

TL;DR

Lightning IR is a user-friendly, modular framework built on PyTorch Lightning that simplifies the integration of transformer-based models into information retrieval pipelines, covering fine-tuning, indexing, searching, and re-ranking.

Contribution

The paper introduces Lightning IR, a scalable, extensible, open-source framework that streamlines the application of transformer models in retrieval tasks.

Findings

Supports all stages of retrieval pipelines

Enhances ease of use and reproducibility

Open-source availability

Abstract

A wide range of transformer-based language models have been proposed for information retrieval tasks. However, including transformer-based models in retrieval pipelines is often complex and requires substantial engineering effort. In this paper, we introduce Lightning IR, an easy-to-use PyTorch Lightning-based framework for applying transformer-based language models in retrieval scenarios. Lightning IR provides a modular and extensible architecture that supports all stages of a retrieval pipeline: from fine-tuning and indexing to searching and re-ranking. Designed to be scalable and reproducible, Lightning IR is available as open-source: https://github.com/webis-de/lightning-ir.