PVO: Panoptic Visual Odometry

TL;DR

PVO is a unified framework that jointly models visual odometry and video panoptic segmentation, improving scene understanding and pose estimation by leveraging mutual benefits between the two tasks.

Contribution

It introduces a panoptic-aware dynamic mask into VO and a fusion-based VPS module, enabling mutual enhancement for more accurate scene modeling.

Findings

PVO outperforms state-of-the-art methods in visual odometry.

PVO achieves superior video panoptic segmentation accuracy.

The mutual optimization improves robustness to dynamic objects.

Abstract

We present PVO, a novel panoptic visual odometry framework to achieve more comprehensive modeling of the scene motion, geometry, and panoptic segmentation information. Our PVO models visual odometry (VO) and video panoptic segmentation (VPS) in a unified view, which makes the two tasks mutually beneficial. Specifically, we introduce a panoptic update module into the VO Module with the guidance of image panoptic segmentation. This Panoptic-Enhanced VO Module can alleviate the impact of dynamic objects in the camera pose estimation with a panoptic-aware dynamic mask. On the other hand, the VO-Enhanced VPS Module also improves the segmentation accuracy by fusing the panoptic segmentation result of the current frame on the fly to the adjacent frames, using geometric information such as camera pose, depth, and optical flow obtained from the VO Module. These two modules contribute to each other through recurrent iterative optimization. Extensive experiments demonstrate that PVO outperforms state-of-the-art methods in both visual odometry and video panoptic segmentation tasks.