Real-Time ESFP: Estimating, Smoothing, Filtering, and Pose-Mapping

TL;DR

This paper introduces ESFP, an end-to-end pipeline that converts monocular RGB videos into accurate, smooth, and anatomically plausible joint trajectories for low-cost robotic arms, integrating estimation, smoothing, filtering, and pose-mapping.

Contribution

The paper presents a novel sequence-to-sequence Transformer-based smoothing module with uncertainty estimation, improving joint trajectory accuracy and plausibility in robotic pose estimation.

Findings

Achieves real-time joint trajectory estimation from monocular video.

Enhances trajectory smoothness and anatomical plausibility.

Demonstrates effective pose retargeting to a low-cost robot.

Abstract

This paper presents ESFP, an end-to-end pipeline that converts monocular RGB video into executable joint trajectories for a low-cost 4-DoF desktop arm. ESFP comprises four sequential modules. (1) Estimating: ROMP lifts each frame to a 24-joint 3-D skeleton. (2) Smoothing: the proposed HPSTM-a sequence-to-sequence Transformer with self-attention-combines long-range temporal context with a differentiable forward-kinematics decoder, enforcing constant bone lengths and anatomical plausibility while jointly predicting joint means and full covariances. (3) Filtering: root-normalized trajectories are variance-weighted according to HPSTM's uncertainty estimates, suppressing residual noise. (4) Pose-Mapping: a geometric retargeting layer transforms shoulder-elbow-wrist triples into the uArm's polar workspace, preserving wrist orientation.