RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedback

TL;DR

RoDiF introduces a robust method for fine-tuning diffusion policies in robotics using corrupted human feedback, combining a unified MDP formulation with a conservative optimization strategy to improve preference alignment and robustness.

Contribution

It presents RoDiF, a novel approach that explicitly handles corrupted human preferences during diffusion policy fine-tuning through a geometric hypothesis-cutting perspective.

Findings

Outperforms state-of-the-art baselines in manipulation tasks

Maintains performance with up to 30% corrupted preferences

Effectively steers policies to human-preferred modes

Abstract

Diffusion policies are a powerful paradigm for robotic control, but fine-tuning them with human preferences is fundamentally challenged by the multi-step structure of the denoising process. To overcome this, we introduce a Unified Markov Decision Process (MDP) formulation that coherently integrates the diffusion denoising chain with environmental dynamics, enabling reward-free Direct Preference Optimization (DPO) for diffusion policies. Building on this formulation, we propose RoDiF (Robust Direct Fine-Tuning), a method that explicitly addresses corrupted human preferences. RoDiF reinterprets the DPO objective through a geometric hypothesis-cutting perspective and employs a conservative cutting strategy to achieve robustness without assuming any specific noise distribution. Extensive experiments on long-horizon manipulation tasks show that RoDiF consistently outperforms state-of-the-art baselines, effectively steering pretrained diffusion policies of diverse architectures to human-preferred modes, while maintaining strong performance even under 30% corrupted preference labels.