# Access Paths for Efficient Ordering with Large Language Models

## TL;DR

This paper introduces the LLM ORDER BY operator, explores its implementations, and develops a dynamic optimizer that improves sorting efficiency and accuracy in large language model systems.

## Contribution

It proposes new semantic sorting algorithms, analyzes their performance, and presents a budget-aware optimizer for selecting optimal access paths in LLM-based sorting.

## Key findings

- No single implementation is universally optimal across datasets.
- A test-time scaling relationship exists between sorting cost and ordering quality.
- The optimizer achieves ranking accuracy comparable or superior to static methods.

## Abstract

In this work, we present the \texttt{LLM ORDER BY} semantic operator as a logical abstraction and conduct a systematic study of its physical implementations. First, we propose several improvements to existing semantic sorting algorithms and introduce a semantic-aware external merge sort algorithm. Our extensive evaluation reveals that no single implementation offers universal optimality on all datasets. From our evaluations, we observe a general test-time scaling relationship between sorting cost and the ordering quality for comparison-based algorithms. Building on these insights, we design a budget-aware optimizer that utilizes heuristic rules, LLM-as-Judge evaluation, and consensus aggregation to dynamically select the near-optimal access path for LLM ORDER BY. In our extensive evaluations, our optimizer consistently achieves ranking accuracy on par with or superior to the best static methods across all benchmarks. We believe that this work provides foundational insights into the principled optimization of semantic operators essential for building robust, large-scale LLM-powered analytic systems.