UNIPO: Unified Interactive Visual Explanation for RL Fine-Tuning Policy Optimization

TL;DR

UNIPO is an interactive visualization tool that unifies the display of token-level training dynamics across various RL fine-tuning algorithms, aiding understanding and comparison.

Contribution

It introduces the first unified, interactive visualization platform for RL fine-tuning algorithms, enhancing accessibility and comparison for non-experts and practitioners.

Findings

Supports classroom instruction for non-experts.

Assists AI practitioners in algorithm selection.

Demonstrates effectiveness through usage scenarios.

Abstract

Reinforcement learning has emerged as a dominant technique for fine-tuning the behavior of large language models, with policy optimization (PO) algorithms such as GRPO, DAPO, and Dr. GRPO emerging in rapid succession to advance state-of-the-art reasoning and alignment performance. However, the modular differences between these algorithms, including targeted improvements to clipping, advantage estimation, and reward aggregation, are introduced across separate papers with inconsistent notation, making them difficult to compare and intimidating to the non-expert community. We present UNIPO, the first interactive visualization tool that exposes the token-level training dynamics of RL fine-tuning algorithms through a unified design. UNIPO connects three complementary views, a high-level training overview, a step-level prompt and response inspector, and a side-by-side algorithm comparison, allowing learners to observe how individual design decisions propagate through training. Through two usage scenarios, we demonstrate how UNIPO supports both classroom instruction for non-experts and algorithm selection for AI practitioners. Our tool is open-source and publicly available at https://poloclub.github.io/unipo.