Comparative Analysis of Lion and AdamW Optimizers for Cross-Encoder Reranking with MiniLM, GTE, and ModernBERT

TL;DR

This study compares the Lion and AdamW optimizers in fine-tuning cross-encoder rerankers across different transformer models and datasets, highlighting Lion's superior effectiveness and efficiency.

Contribution

It introduces a comprehensive analysis of Lion optimizer's impact on cross-encoder reranking models, demonstrating its advantages over AdamW in effectiveness and GPU utilization.

Findings

Lion optimizer improves NDCG@10 and MAP scores.

Lion enhances GPU efficiency by up to 10.33%.

ModernBERT with Lion achieves top performance on TREC DL 2019.

Abstract

Modern information retrieval systems often employ a two-stage pipeline: an efficient initial retrieval stage followed by a computationally intensive reranking stage. Cross-encoders have shown strong effectiveness for reranking due to their deep analysis of query-document pairs. This paper studies the impact of the Lion optimizer, a recent alternative to AdamW, during fine-tuning of cross-encoder rerankers. We fine-tune three transformer models-MiniLM, GTE, and ModernBERT-on the MS MARCO passage ranking dataset using both optimizers. GTE and ModernBERT support extended context lengths (up to 8192 tokens). We evaluate effectiveness using TREC 2019 Deep Learning Track and MS MARCO dev set (MRR@10). Experiments, run on the Modal cloud platform, reveal that ModernBERT with Lion achieves the best NDCG@10 (0.7225) and MAP (0.5121) on TREC DL 2019, while MiniLM with Lion ties ModernBERT for MRR@10 (0.5988) on MS MARCO dev. Lion also provides superior GPU efficiency, improving utilization by 2.67% to 10.33% across models. We analyze performance trends using standard IR metrics and discuss the optimizer's impact on training dynamics across architectures.