MCTS with Refinement for Proposals Selection Games in Scene Understanding

TL;DR

This paper introduces a novel MCTS-based method for scene understanding that jointly selects and refines proposals, demonstrated on floor plan reconstruction and 3D room layout estimation, achieving state-of-the-art results.

Contribution

It adapts Monte Carlo Tree Search for proposal selection and refinement in scene understanding, including a differentiable renderer for polygonal shapes, applicable to 2D and 3D layout reconstruction.

Findings

Significant improvements over state-of-the-art on Structured3D and Floor-SP datasets.

Effective joint proposal selection and optimization without hard constraints.

High accuracy in 3D room layout reconstruction from images.

Abstract

We propose a novel method applicable in many scene understanding problems that adapts the Monte Carlo Tree Search (MCTS) algorithm, originally designed to learn to play games of high-state complexity. From a generated pool of proposals, our method jointly selects and optimizes proposals that minimize the objective term. In our first application for floor plan reconstruction from point clouds, our method selects and refines the room proposals, modelled as 2D polygons, by optimizing on an objective function combining the fitness as predicted by a deep network and regularizing terms on the room shapes. We also introduce a novel differentiable method for rendering the polygonal shapes of these proposals. Our evaluations on the recent and challenging Structured3D and Floor-SP datasets show significant improvements over the state-of-the-art, without imposing hard constraints nor assumptions on the floor plan configurations. In our second application, we extend our approach to reconstruct general 3D room layouts from a color image and obtain accurate room layouts. We also show that our differentiable renderer can easily be extended for rendering 3D planar polygons and polygon embeddings. Our method shows high performance on the Matterport3D-Layout dataset, without introducing hard constraints on room layout configurations.