ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise Queries

TL;DR

ExPairT-LLM introduces an exact learning algorithm for code selection that uses pairwise queries to an LLM oracle, improving accuracy and robustness over existing methods in selecting correct code snippets.

Contribution

The paper proposes a novel pairwise query-based learning algorithm for code selection that is more accurate and robust than prior approaches.

Findings

Pass@1 success rate improves by up to 27.1% over state-of-the-art.

Method enhances LLMs' performance on complex reasoning tasks.

Robustness to some LLM mistakes demonstrated in experiments.

Abstract

Despite recent advances in LLMs, the task of code generation is still challenging. To cope, code selection algorithms select the best program from multiple programs generated by an LLM. However, existing algorithms can fail to identify the correct program, either because they can misidentify nonequivalent programs or because they rely on an LLM and assume it always correctly determines the output for every input. We present ExPairT-LLM, an exact learning algorithm for code selection that selects a program by posing to an LLM oracle two new types of queries: pairwise membership and pairwise equivalence. These queries are simpler for LLMs and enable ExPairT-LLM to identify the correct program through a tournament, which is robust to some LLM mistakes. We evaluate ExPairT-LLM on four popular code datasets. Its pass@1 (success rate) outperforms the state-of-the-art code selection algorithm on average by +13.0% and up to +27.1%. It also improves the pass@1 of LLMs performing complex reasoning by +24.0%.