Non-myopic Generation of Language Models for Reasoning and Planning

TL;DR

This paper introduces Predictive-Decoding, a novel approach that applies Model Predictive Control to Large Language Models, significantly improving their reasoning and planning accuracy while reducing computational costs.

Contribution

It proposes a new method, Predictive-Decoding, that enhances LLM planning by incorporating foresight and optimal-control principles, addressing their inherent myopic limitations.

Findings

Improved reasoning and planning accuracy across diverse tasks.

Enhanced computational efficiency over search baselines.

Effective mitigation of early errors in LLM planning.

Abstract

Large Language Models have demonstrated remarkable abilities in reasoning and planning by breaking down complex problems into sequential steps. Despite their success in various domains like mathematical problem-solving and coding, LLMs face challenges in ensuring reliable and optimal planning due to their inherent myopic nature of autoregressive decoding. This paper revisits LLM reasoning from an optimal-control perspective, proposing a novel method, Predictive-Decoding, that leverages Model Predictive Control to enhance planning accuracy. By re-weighting LLM distributions based on foresight trajectories, Predictive-Decoding aims to mitigate early errors and promote non-myopic planning. Our experiments show significant improvements in a wide range of tasks for math, coding, and agents. Furthermore, Predictive-Decoding demonstrates computational efficiency, outperforming search baselines with reduced computational resources. This study provides insights into optimizing LLM planning capabilities.