Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking

TL;DR

This paper introduces TRACE, a turn-aware credit assignment framework for multi-turn jailbreaking of LLMs, improving attack success and safety-utility balance by addressing the non-uniform contribution of dialogue turns.

Contribution

The paper proposes a novel turn-level credit assignment method for RL-based multi-turn jailbreaking, enhancing effectiveness and safety in attack and defense scenarios.

Findings

TRACE achieves about 25% relative improvement in attack success rate.

It improves safety-utility balance when reused for defense.

Extensive experiments validate TRACE's effectiveness, transferability, and efficiency.

Abstract

Deploying LLMs in multi-turn dialogues facilitates jailbreak attacks that distribute harmful intent across seemingly benign turns. Recent training-based multi-turn jailbreak methods learn long-horizon attack strategies from interaction feedback, but often rely on coarse trajectory-level outcome signals that broadcast uniformly to every turn. However, we find that turn-level contributions in multi-turn jailbreaking are non-uniform, phase-dependent, and target-specific. Such coarse outcome supervision induces a credit assignment problem, leading to over-rewarding redundant turns in successful trajectories and under-crediting useful intermediate turns in failed ones. To address this, we propose TRACE, a turn-aware credit assignment framework for reinforcement learning (RL)-based multi-turn jailbreaking. For successful trajectories, TRACE estimates turn-level contributions via leave-one-turn-out semantic masking; for failed ones, TRACE assigns penalties based on prompt harmfulness and semantic relevance, with an additional local refusal-aware penalty. Furthermore, we reuse the attack-side credit signal for multi-turn defense alignment. Extensive experiments on open-source and closed-source targets show that TRACE achieves strong overall performance in effectiveness, transferability, and efficiency, yielding about a 25% relative improvement in attack success rate over the strongest RL baseline while also improving the safety-utility balance when reused for defense alignment.