Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively

TL;DR

This paper introduces the Speculative Reward Model (SRM), a cost-effective framework for improving LLM decision-making by reducing computational costs while maintaining high performance across complex tasks.

Contribution

The paper proposes SRM, a novel plug-and-play framework that enhances LLM decision-making efficiency by integrating an external reward predictor and speculative verification, balancing effectiveness and cost.

Findings

SRM reduces search costs to 1/10 of traditional methods on average.

SRM maintains high decision-making effectiveness across various tasks.

The 3E Criteria effectively assess search strategy cost-effectiveness.

Abstract

Effective decision-making in Large Language Models (LLMs) is essential for handling intricate tasks. However, existing approaches prioritize performance but often overlook the balance between effectiveness and computational cost. To address this, we first introduce the 3E Criteria to systematically assess the cost-effectiveness of search strategies, revealing that existing methods often trade significant efficiency for marginal performance gains. To improve LLM decision-making while maintaining efficiency, we propose the Speculative Reward Model (SRM), a plug-and-play framework that seamlessly integrates with existing search strategies. Specifically, SRM employs an external reward assigner to predict optimal actions, reducing reliance on LLMs' internal self-evaluation. And a speculative verification mechanism is used to prune suboptimal choices and guide the search toward more promising steps. We evaluate SRM on several complex decision-making tasks including mathematical reasoning, planning and numerical reasoning in specialized domains. Experimental results show that SRM reduces costs to 1/10 of the original search framework on average while maintaining effectiveness.