Physically Plausible Pose Refinement using Fully Differentiable Forces

TL;DR

This paper introduces a fully differentiable force-based model for refining hand-object poses from mesh data, improving contact accuracy and resolving mesh issues without relying on RGB or depth images.

Contribution

It presents a novel end-to-end differentiable framework that estimates forces to refine pose and contact predictions, addressing limitations of previous methods.

Findings

Successfully refines poses and contact maps on ContactPose dataset

Improves contact accuracy without using RGB or depth data

Resolves mesh interpenetration issues

Abstract

All hand-object interaction is controlled by forces that the two bodies exert on each other, but little work has been done in modeling these underlying forces when doing pose and contact estimation from RGB/RGB-D data. Given the pose of the hand and object from any pose estimation system, we propose an end-to-end differentiable model that refines pose estimates by learning the forces experienced by the object at each vertex in its mesh. By matching the learned net force to an estimate of net force based on finite differences of position, this model is able to find forces that accurately describe the movement of the object, while resolving issues like mesh interpenetration and lack of contact. Evaluating on the ContactPose dataset, we show this model successfully corrects poses and finds contact maps that better match the ground truth, despite not using any RGB or depth image data.