Are Optimal Algorithms Still Optimal? Rethinking Sorting in LLM-Based Pairwise Ranking with Batching and Caching

TL;DR

This paper reevaluates sorting algorithms in LLM-based pairwise ranking by focusing on inference costs, revealing that classical optimal algorithms may become inefficient when batching and caching are employed to reduce inference expenses.

Contribution

It introduces a new framework that shifts the cost analysis from comparisons to LLM inferences, highlighting the impact of batching and caching on algorithm efficiency.

Findings

Classical optimal algorithms may be less efficient with LLM inference costs.

Batching and caching strategies significantly reduce inference expenses.

Traditional comparison-based metrics are insufficient for LLM-based ranking efficiency.

Abstract

We introduce a novel framework for analyzing sorting algorithms in pairwise ranking prompting (PRP), re-centering the cost model around LLM inferences rather than traditional pairwise comparisons. While classical metrics based on comparison counts have traditionally been used to gauge efficiency, our analysis reveals that expensive LLM inferences overturn these predictions; accordingly, our framework encourages strategies such as batching and caching to mitigate inference costs. We show that algorithms optimal in the classical setting can lose efficiency when LLM inferences dominate the cost under certain optimizations.