PQ-Transformer: Jointly Parsing 3D Objects and Layouts from Point Clouds

TL;DR

The paper introduces PQ-Transformer, a novel neural network architecture that jointly predicts 3D objects and room layouts from point clouds, improving efficiency and accuracy in 3D scene understanding for robotics.

Contribution

It presents the first transformer-based model for simultaneous 3D object and layout parsing from point clouds, with a new quad-based layout representation and a physical constraint loss.

Findings

Achieves near real-time performance at 8.91 FPS.

Significantly improves room layout F1-score from 37.9% to 57.9%.

Outperforms existing methods on the ScanNet benchmark.

Abstract

3D scene understanding from point clouds plays a vital role for various robotic applications. Unfortunately, current state-of-the-art methods use separate neural networks for different tasks like object detection or room layout estimation. Such a scheme has two limitations: 1) Storing and running several networks for different tasks are expensive for typical robotic platforms. 2) The intrinsic structure of separate outputs are ignored and potentially violated. To this end, we propose the first transformer architecture that predicts 3D objects and layouts simultaneously, using point cloud inputs. Unlike existing methods that either estimate layout keypoints or edges, we directly parameterize room layout as a set of quads. As such, the proposed architecture is termed as P(oint)Q(uad)-Transformer. Along with the novel quad representation, we propose a tailored physical constraint loss function that discourages object-layout interference. The quantitative and qualitative evaluations on the public benchmark ScanNet show that the proposed PQ-Transformer succeeds to jointly parse 3D objects and layouts, running at a quasi-real-time (8.91 FPS) rate without efficiency-oriented optimization. Moreover, the new physical constraint loss can improve strong baselines, and the F1-score of the room layout is significantly promoted from 37.9% to 57.9%.