Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration

TL;DR

Soft Reasoning introduces an embedding-based search framework for large language models that enhances reasoning accuracy and coherence by controlled exploration and Bayesian optimisation, without heuristic search.

Contribution

It presents a novel embedding exploration method combining perturbation and Bayesian optimisation to improve LLM reasoning capabilities.

Findings

Achieves higher reasoning accuracy than baseline methods.

Requires minimal additional computation.

Proven effective across multiple models and tasks.

Abstract

Large Language Models (LLMs) struggle with complex reasoning due to limited diversity and inefficient search. We propose Soft Reasoning, an embedding-based search framework that optimises the embedding of the first token to guide generation. It combines (1) embedding perturbation for controlled exploration and (2) Bayesian optimisation to refine embeddings via a verifier-guided objective, balancing exploration and exploitation. This approach improves reasoning accuracy and coherence while avoiding reliance on heuristic search. Experiments demonstrate superior correctness with minimal computation, making it a scalable, model-agnostic solution. The code is released at https://github.com/alickzhu/Soft-Reasoning.