Foresight Optimization for Strategic Reasoning in Large Language Models

TL;DR

This paper introduces Foresight Policy Optimization (FoPO), a novel method to improve strategic reasoning in large language models by explicitly modeling opponent behavior and foresight, leading to better decision-making in multi-agent scenarios.

Contribution

The paper proposes FoPO, integrating opponent modeling into policy optimization, and creates datasets to systematically evaluate strategic reasoning in LLMs.

Findings

FoPO significantly improves strategic reasoning across various LLM sizes.

Models trained with FoPO generalize well to out-of-domain strategic scenarios.

FoPO outperforms standard reasoning optimization baselines.

Abstract

Reasoning capabilities in large language models (LLMs) have generally advanced significantly. However, it is still challenging for existing reasoning-based LLMs to perform effective decision-making abilities in multi-agent environments, due to the absence of explicit foresight modeling. To this end, strategic reasoning, the most fundamental capability to anticipate the counterpart's behaviors and foresee its possible future actions, has been introduced to alleviate the above issues. Strategic reasoning is fundamental to effective decision-making in multi-agent environments, yet existing reasoning enhancement methods for LLMs do not explicitly capture its foresight nature. In this work, we introduce Foresight Policy Optimization (FoPO) to enhance strategic reasoning in LLMs, which integrates opponent modeling principles into policy optimization, thereby enabling explicit consideration of both self-interest and counterpart influence. Specifically, we construct two curated datasets, namely Cooperative RSA and Competitive Taboo, equipped with well-designed rules and moderate difficulty to facilitate a systematic investigation of FoPO in a self-play framework. Our experiments demonstrate that FoPO significantly enhances strategic reasoning across LLMs of varying sizes and origins. Moreover, models trained with FoPO exhibit strong generalization to out-of-domain strategic scenarios, substantially outperforming standard LLM reasoning optimization baselines.