In-Context Reinforcement Learning for Tool Use in Large Language Models

TL;DR

This paper introduces In-Context Reinforcement Learning (ICRL), a novel RL-only approach that enables large language models to learn tool use without supervised fine-tuning, improving reasoning and factual retrieval capabilities efficiently.

Contribution

ICRL eliminates the need for supervised fine-tuning by using few-shot prompts during RL, gradually reducing in-context examples to enable zero-shot tool invocation in large language models.

Findings

ICRL achieves state-of-the-art results on reasoning benchmarks.

ICRL is more data-efficient than traditional fine-tuning methods.

Models trained with ICRL can call external tools independently in zero-shot settings.

Abstract

While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.