Unleashing the Creative Mind: Language Model As Hierarchical Policy For Improved Exploration on Challenging Problem Solving

TL;DR

This paper introduces a hierarchical policy framework for large language models that enhances exploration of diverse problem-solving strategies, leading to improved accuracy on challenging reasoning tasks by using a visionary leader and a detailed follower with a tournament-based selection.

Contribution

It proposes framing LLMs as hierarchical policies with high-level hints and detailed execution, significantly improving exploration and solution accuracy on complex problems.

Findings

Enhanced exploration of diverse strategies in LLMs.

Improved accuracy on challenging math problems.

Effective tournament-based solution selection.

Abstract

Large Language Models (LLMs) have achieved tremendous progress, yet they still often struggle with challenging reasoning problems. Current approaches address this challenge by sampling or searching detailed and low-level reasoning chains. However, these methods are still limited in their exploration capabilities, making it challenging for correct solutions to stand out in the huge solution space. In this work, we unleash LLMs' creative potential for exploring multiple diverse problem solving strategies by framing an LLM as a hierarchical policy via in-context learning. This policy comprises of a visionary leader that proposes multiple diverse high-level problem-solving tactics as hints, accompanied by a follower that executes detailed problem-solving processes following each of the high-level instruction. The follower uses each of the leader's directives as a guide and samples multiple reasoning chains to tackle the problem, generating a solution group for each leader proposal. Additionally, we propose an effective and efficient tournament-based approach to select among these explored solution groups to reach the final answer. Our approach produces meaningful and inspiring hints, enhances problem-solving strategy exploration, and improves the final answer accuracy on challenging problems in the MATH dataset. Code will be released at https://github.com/lz1oceani/LLM-As-Hierarchical-Policy.