CRISP: Object Pose and Shape Estimation with Test-Time Adaptation

TL;DR

CRISP is a novel, category-agnostic pipeline for object pose and shape estimation from RGB-D images that incorporates test-time adaptation and self-supervised domain adaptation to improve accuracy across diverse datasets and unseen objects.

Contribution

Introduces CRISP, a category-agnostic object pose and shape estimation pipeline with a shape correction method and a self-training domain adaptation approach.

Findings

High performance on YCBV, SPE3R, and NOCS datasets.

Effective bridging of large domain gaps through self-training.

Ability to generalize to unseen objects.

Abstract

We consider the problem of estimating object pose and shape from an RGB-D image. Our first contribution is to introduce CRISP, a category-agnostic object pose and shape estimation pipeline. The pipeline implements an encoder-decoder model for shape estimation. It uses FiLM-conditioning for implicit shape reconstruction and a DPT-based network for estimating pose-normalized points for pose estimation. As a second contribution, we propose an optimization-based pose and shape corrector that can correct estimation errors caused by a domain gap. Observing that the shape decoder is well behaved in the convex hull of known shapes, we approximate the shape decoder with an active shape model, and show that this reduces the shape correction problem to a constrained linear least squares problem, which can be solved efficiently by an interior point algorithm. Third, we introduce a self-training pipeline to perform self-supervised domain adaptation of CRISP. The self-training is based on a correct-and-certify approach, which leverages the corrector to generate pseudo-labels at test time, and uses them to self-train CRISP. We demonstrate CRISP (and the self-training) on YCBV, SPE3R, and NOCS datasets. CRISP shows high performance on all the datasets. Moreover, our self-training is capable of bridging a large domain gap. Finally, CRISP also shows an ability to generalize to unseen objects. Code and pre-trained models will be available on https://web.mit.edu/sparklab/research/crisp_object_pose_shape/.