A Compute-Matched Re-Evaluation of TroVE on MATH

TL;DR

This paper re-evaluates TroVE's effectiveness on the MATH benchmark, finding that its apparent benefits are mainly due to increased computational resources rather than its toolbox mechanism.

Contribution

It demonstrates that TroVE's advantage over baseline methods is primarily from higher compute, and not from its toolbox or reuse strategies, after correcting and controlling for compute.

Findings

TroVE's performance gain is largely due to more compute used.

Correcting TroVE's selection mechanism improves accuracy by 3%.

After compute matching, TroVE's advantage drops to 1%.

Abstract

Reusing established theorems and formulas is central to mathematical problem solving, serving as essential building blocks for tackling increasingly complex challenges. Recent work, TroVE, argues that code-generating Large Language Models (LLMs) can benefit similarly on the MATH benchmark by inducing and reusing higher-level toolboxes. By allocating computational budget across an ensemble of three modes -- directly generating code, creating tools, and reusing tools -- TroVE claims to outperform a PRIMITIVE baseline that only performs direct generation. However, recent analysis (Berlot-Attwell et al., 2024) casts doubt on these gains, noting that the tools created are often trivial or rarely reused, suggesting that improvements may stem from self-consistency or self-correction. In this work, we re-evaluate TroVE on MATH, analyze the impact of each of its modes, and show that its benefit does not come from these mechanisms, but simply from a higher computational budget spent for TroVE compared to PRIMITIVE. To this end, we also perform a small correction in the original implementation of TroVE's selection mechanism, boosting TroVE's performance on MATH by 3\% in accuracy. After matching for compute, the benefit of TroVE reduces to a marginal improvement of 1\%, suggesting that this toolbox approach does not provide a significant benefit on MATH.