Keyframe-Guided Structured Rewards for Reinforcement Learning in Long-Horizon Laboratory Robotics

TL;DR

This paper introduces a keyframe-guided reward framework for reinforcement learning that improves long-horizon laboratory robotic tasks by automatically extracting keyframes and generating structured rewards, leading to higher success rates.

Contribution

The paper presents a novel reward generation framework that leverages keyframes and diffusion-based predictors to enhance reinforcement learning in complex laboratory automation tasks.

Findings

Achieved 82% success rate after 40-60 minutes of fine-tuning.

Outperformed existing methods HG-DAgger and Hil-ConRFT.

Effective in high-precision pipette attachment and liquid transfer tasks.

Abstract

Long-horizon precision manipulation in laboratory automation, such as pipette tip attachment and liquid transfer, requires policies that respect strict procedural logic while operating in continuous, high-dimensional state spaces. However, existing approaches struggle with reward sparsity, multi-stage structural constraints, and noisy or imperfect demonstrations, leading to inefficient exploration and unstable convergence. We propose a Keyframe-Guided Reward Generation Framework that automatically extracts kinematics-aware keyframes from demonstrations, generates stage-wise targets via a diffusion-based predictor in latent space, and constructs a geometric progress-based reward to guide online reinforcement learning. The framework integrates multi-view visual encoding, latent similarity-based progress tracking, and human-in-the-loop reinforcement fine-tuning on a Vision-Language-Action backbone to align policy optimization with the intrinsic stepwise logic of biological protocols. Across four real-world laboratory tasks, including high-precision pipette attachment and dynamic liquid transfer, our method achieves an average success rate of 82% after 40--60 minutes of online fine-tuning. Compared with HG-DAgger (42%) and Hil-ConRFT (47%), our approach demonstrates the effectiveness of structured keyframe-guided rewards in overcoming exploration bottlenecks and providing a scalable solution for high-precision, long-horizon robotic laboratory automation.