BOP-Distrib: Revisiting 6D Pose Estimation Benchmarks for Better Evaluation under Visual Ambiguities

TL;DR

This paper introduces a new method for re-annotating 6D pose datasets with image-specific pose distributions, leading to more accurate benchmarking of pose estimation methods under visual ambiguities.

Contribution

It proposes an automatic re-annotation process for datasets, and a new evaluation framework for pose distribution methods, improving the assessment of 6D pose estimation accuracy.

Findings

Re-annotated datasets reveal different method rankings.

Improved ground truth affects evaluation outcomes.

Benchmark for pose distribution methods on real images is established.

Abstract

6D pose estimation aims at determining the object pose that best explains the camera observation. The unique solution for non-ambiguous objects can turn into a multi-modal pose distribution for symmetrical objects or when occlusions of symmetry-breaking elements happen, depending on the viewpoint. Currently, 6D pose estimation methods are benchmarked on datasets that consider, for their ground truth annotations, visual ambiguities as only related to global object symmetries, whereas they should be defined per-image to account for the camera viewpoint. We thus first propose an automatic method to re-annotate those datasets with a 6D pose distribution specific to each image, taking into account the object surface visibility in the image to correctly determine the visual ambiguities. Second, given this improved ground truth, we re-evaluate the state-of-the-art single pose methods and show that this greatly modifies the ranking of these methods. Third, as some recent works focus on estimating the complete set of solutions, we derive a precision/recall formulation to evaluate them against our image-wise distribution ground truth, making it the first benchmark for pose distribution methods on real images.