CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance

TL;DR

CodeSteer introduces a novel method for guiding large language models with symbolic and multi-turn guidance, significantly improving their performance on complex symbolic tasks and enabling better symbolic reasoning capabilities.

Contribution

The paper presents CodeSteer, a new approach combining symbolic guidance with fine-tuning techniques to enhance LLMs' ability to handle symbolic reasoning and code generation tasks.

Findings

CodeSteerLLM outperforms existing models on 37 symbolic tasks.

Augmenting GPT-4o with CodeSteer raises its performance from 53.3 to 86.4.

CodeSteer enhances generalization across different LLMs and unseen tasks.

Abstract

Existing methods fail to effectively steer Large Language Models (LLMs) between textual reasoning and code generation, leaving symbolic computing capabilities underutilized. We introduce CodeSteer, an effective method for guiding LLM code/text generation. We construct a comprehensive benchmark SymBench comprising 37 symbolic tasks with adjustable complexity and also synthesize datasets of 12k multi-turn guidance/generation trajectories and 5.5k guidance comparison pairs. We fine-tune the Llama-3-8B model with a newly designed multi-turn supervised fine-tuning (SFT) and direct preference optimization (DPO). The resulting model, CodeSteerLLM, augmented with the proposed symbolic and self-answer checkers, effectively guides the code/text generation of larger models. Augmenting GPT-4o with CodeSteer raises its average performance score from 53.3 to 86.4, even outperforming the existing best LLM OpenAI o1 (82.7), o1-preview (74.8), and DeepSeek R1 (76.8) across all 37 tasks (28 seen, 9 unseen). Trained for GPT-4o, CodeSteer demonstrates superior generalizability, providing an average 41.8 performance boost on Claude, Mistral, and GPT-3.5. CodeSteer-guided LLMs fully harness symbolic computing to maintain strong performance on highly complex tasks. Models, Datasets, and Codes are available at https://github.com/yongchao98/CodeSteer-v1.0 and https://huggingface.co/yongchao98.