Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents

TL;DR

This paper introduces DuCA, a novel reinforcement learning framework that disentangles and balances long-term and immediate goals in industrial sales language models, leading to improved conversion and language quality.

Contribution

The paper proposes Dual-Horizon Credit Assignment (DuCA) with Horizon-Independent Advantage Normalization (HIAN), a new method for balancing heterogeneous rewards in multi-turn RL for sales agents.

Findings

Achieves 6.82% relative improvement in conversion rate.

Reduces inter-sentence repetition by 82.28%.

Lowers identity detection rate by 27.35%.

Abstract

Optimizing large language models for industrial sales requires balancing long-term commercial objectives (e.g., conversion rate) with immediate linguistic constraints such as fluency and compliance. Conventional reinforcement learning often merges these heterogeneous goals into a single reward, causing high-magnitude session-level rewards to overwhelm subtler turn-level signals, which leads to unstable training or reward hacking. To address this issue, we propose Dual-Horizon Credit Assignment (DuCA), a framework that disentangles optimization across time scales. Its core, Horizon-Independent Advantage Normalization (HIAN), separately normalizes advantages from turn-level and session-level rewards before fusion, ensuring balanced gradient contributions from both immediate and long-term objectives to the policy update. Extensive experiments with a high-fidelity user simulator show DuCA outperforms the state-of-the-art GRPO baseline, achieving a 6.82% relative improvement in conversion rate, reducing inter-sentence repetition by 82.28%, and lowering identity detection rate by 27.35%, indicating a substantial improvement for an industrial sales scenario that effectively balances the dual demands of strategic performance and naturalistic language generation.