Training with Pseudo-Code for Instruction Following

TL;DR

This paper introduces a training method that enhances large language models' ability to follow instructions by incorporating pseudo-code representations during fine-tuning, leading to significant improvements across various benchmarks.

Contribution

The authors propose a novel instruction-tuning approach using pseudo-code data, improving instruction-following performance without sacrificing reasoning capabilities.

Findings

8-21% improvement on instruction-following benchmarks

Up to 30% average gain across multiple tasks

Models maintain or improve reasoning performance

Abstract

Despite rapid advances in the capabilities of Large Language Models (LLMs), they continue to struggle with following relatively simple and unambiguous instructions, particularly when compositional structure is involved. Recent work suggests that models may follow instructions more effectively when they are expressed in pseudo-code rather than natural language. However, writing pseudo-code programs can be tedious, and relying on few-shot demonstrations or inference-time code prompting is often unnatural for non-expert users of LLMs. To overcome these limitations, we propose a training time approach that fine-tunes LLMs using instruction-tuning data augmented with pseudo-code representations of natural language instructions paired with final responses. We evaluate our method on 12 publicly available benchmarks spanning instruction-following, mathematical reasoning, and commonsense reasoning, across six base models. Our results show that models trained with pseudo-code follow instructions more reliably, achieving relative gains of 8-21\% on instruction following benchmarks, while largely preserving and in some cases improving performance on mathematical and commonsense reasoning tasks, with an average gain of up to 30\% across all evaluated benchmarks.