SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows

TL;DR

SERFN introduces a sample-efficient, likelihood-based fine-tuning framework for dexterous manipulation, combining normalizing flows and chunked critics to handle multimodal actions and improve real-world robotic performance.

Contribution

The paper presents a novel off-policy fine-tuning method using normalizing flows and chunked critics, enabling stable, sample-efficient dexterous policy adaptation on real robots.

Findings

SERFN achieves stable, sample-efficient fine-tuning on real robotic tasks.

Normalizing flows enable exact likelihood computation for multimodal action chunks.

Chunked critics improve long-horizon credit assignment in dexterous manipulation.

Abstract

Real-world fine-tuning of dexterous manipulation policies remains challenging due to limited real-world interaction budgets and highly multimodal action distributions. Diffusion-based policies, while expressive, do not permit conservative likelihood-based updates during fine-tuning because action probabilities are intractable. In contrast, conventional Gaussian policies collapse under multimodality, particularly when actions are executed in chunks, and standard per-step critics fail to align with chunked execution, leading to poor credit assignment. We present SERFN, a sample-efficient off-policy fine-tuning framework with normalizing flow (NF) to address these challenges. The normalizing flow policy yields exact likelihoods for multimodal action chunks, allowing conservative, stable policy updates through likelihood regularization and thereby improving sample efficiency. An action-chunked critic evaluates entire action sequences, aligning value estimation with the policy's temporal structure and improving long-horizon credit assignment. To our knowledge, this is the first demonstration of a likelihood-based, multimodal generative policy combined with chunk-level value learning on real robotic hardware. We evaluate SERFN on two challenging dexterous manipulation tasks in the real world: cutting tape with scissors retrieved from a case, and in-hand cube rotation with a palm-down grasp -- both of which require precise, dexterous control over long horizons. On these tasks, SERFN achieves stable, sample-efficient adaptation where standard methods struggle.