The Reasoning-Creativity Trade-off: Toward Creativity-Driven Problem Solving

TL;DR

This paper introduces a new framework called Distributional Creative Reasoning (DCR) to address the trade-off between correctness and creativity in large language models, providing methods to prevent reasoning collapse and promote diverse, creative solutions.

Contribution

The paper proposes DCR, a unified variational objective, and offers theoretical insights and practical recipes to maintain both correctness and creativity in LLM reasoning.

Findings

DCR unifies various training methods under a common framework.

The diversity decay theorem explains how correctness objectives cause diversity loss.

Practical recipes are provided to prevent reasoning collapse in LLMs.

Abstract

State-of-the-art large language model (LLM) pipelines rely on bootstrapped reasoning loops: sampling diverse chains of thought and reinforcing the highest-scoring ones, mainly optimizing correctness. We analyze how this design choice is sensitive to the collapse of the model's distribution over reasoning paths, slashing semantic entropy and undermining creative problem-solving. To analyze this failure, we introduce Distributional Creative Reasoning (DCR), a unified variational objective that casts training as gradient flow through probability measures on solution traces. STaR, GRPO, and DPO, as well as entropy bonuses, and other methods, all constitute special cases of the same loss. The framework delivers three core results: (i) the diversity decay theorem, describing how correctness-based objectives lead to distinct modes of diversity decay for STaR, GRPO, and DPO; (ii) designs that ensure convergence to a stable and diverse policy, effectively preventing collapse; and (iii) simple, actionable recipes to achieve this in practice. DCR thus offers the first principled recipe for LLMs that remain both correct and creative.