MOTIF: Modular Thinking via Reinforcement Fine-tuning in LLMs

TL;DR

MOTIF introduces a reinforcement learning fine-tuning approach enabling large language models to perform modular, multi-round reasoning beyond their context size limits, improving accuracy efficiently.

Contribution

The paper presents MOTIF, a novel RL fine-tuning method that enhances LLM reasoning by enabling multi-round thinking over larger contexts, with improved accuracy and sample efficiency.

Findings

Achieved 3.8% and 3.3% accuracy improvements on benchmarks.

Demonstrated effective reasoning beyond context size limits.

Sample-efficient training with only 15% of data.

Abstract

Recent advancements in the reasoning capabilities of large language models (LLMs) show that employing group relative policy optimization (GRPO) algorithm for reinforcement learning (RL) training allows the models to use more thinking/reasoning tokens for generating better responses. However, LLMs can generate only a finite amount of tokens while maintaining attention to the previously generated tokens. This limit, also known as the context size of an LLM, is a bottleneck in LLM reasoning with arbitrarily large number of tokens. To think beyond the limit of context size, an LLM must employ a modular thinking strategy to reason over multiple rounds. In this work, we propose $\textbf{MOTIF: Modular Thinking via Reinforcement Finetuning}$ -- an RL training method for generating thinking tokens in multiple rounds, effectively allowing the model to think with additional context size. We trained the open-source model Qwen2.5-3B-Instruct on GSM8K dataset via parameter efficient fine-tuning and tested its accuracy on MATH500 and AIME2024 benchmarks. Our experiments show 3.8\% and 3.3\% improvements over vanilla GRPO based training in the respective benchmarks. Furthermore, this improvement was achieved with only 15\% of samples, thus demonstrating sample efficiency of MOTIF. Our code and models are available at https://github.com/purbeshmitra/MOTIF and https://huggingface.co/purbeshmitra/MOTIF, respectively.