When Actions Teach You to Think: Reasoning-Action Synergy via Reinforcement Learning in Conversational Agents

TL;DR

This paper introduces a reinforcement learning approach that enables conversational agents to learn reasoning strategies directly from task outcomes, improving reasoning quality and tool use without relying on costly annotated reasoning traces.

Contribution

It presents a novel pipeline where LLMs generate reasoning steps to guide tool invocation and answer generation, optimized with Group Relative Policy Optimization for better generalization.

Findings

Achieves 1.5% improvement over non-reasoning SFT models

Attains 40% gain over baseline Qwen3-1.7B model

Enhances reasoning quality and tool invocation accuracy

Abstract

Supervised fine-tuning (SFT) has emerged as one of the most effective ways to improve the performance of large language models (LLMs) in downstream tasks. However, SFT can have difficulty generalizing when the underlying data distribution changes, even when the new data does not fall completely outside the training domain. Recent reasoning-focused models such as o1 and R1 have demonstrated consistent gains over their non-reasoning counterparts, highlighting the importance of reasoning for improved generalization and reliability. However, collecting high-quality reasoning traces for SFT remains challenging -- annotations are costly, subjective, and difficult to scale. To address this limitation, we leverage Reinforcement Learning (RL) to enable models to learn reasoning strategies directly from task outcomes. We propose a pipeline in which LLMs generate reasoning steps that guide both the invocation of tools (e.g., function calls) and the final answer generation for conversational agents. Our method employs Group Relative Policy Optimization (GRPO) with rewards designed around tool accuracy and answer correctness, allowing the model to iteratively refine its reasoning and actions. Experimental results demonstrate that our approach improves both the quality of reasoning and the precision of tool invocations, achieving a 1.5% relative improvement over the SFT model (trained without explicit thinking) and a 40% gain compared to the base of the vanilla Qwen3-1.7B model. These findings demonstrate the promise of unifying reasoning and action learning through RL to build more capable and generalizable conversational agents.