PolyDiffuse: Polygonal Shape Reconstruction via Guided Set Diffusion Models

TL;DR

PolyDiffuse introduces a guided set diffusion model for reconstructing polygonal shapes from sensor data, effectively handling the ambiguity of set representations and enabling accurate, conditional shape generation for applications like floorplans and HD maps.

Contribution

The paper proposes a novel Guided Set Diffusion Model that controls noise injection to resolve set permutation ambiguity and reconstructs polygonal shapes conditioned on sensor data.

Findings

Significantly outperforms existing methods on benchmark datasets.

Effectively reconstructs complex polygonal shapes from sensor data.

Enables practical applications in autonomous driving and architectural design.

Abstract

This paper presents PolyDiffuse, a novel structured reconstruction algorithm that transforms visual sensor data into polygonal shapes with Diffusion Models (DM), an emerging machinery amid exploding generative AI, while formulating reconstruction as a generation process conditioned on sensor data. The task of structured reconstruction poses two fundamental challenges to DM: 1) A structured geometry is a ``set'' (e.g., a set of polygons for a floorplan geometry), where a sample of $N$ elements has $N!$ different but equivalent representations, making the denoising highly ambiguous; and 2) A ``reconstruction'' task has a single solution, where an initial noise needs to be chosen carefully, while any initial noise works for a generation task. Our technical contribution is the introduction of a Guided Set Diffusion Model where 1) the forward diffusion process learns guidance networks to control noise injection so that one representation of a sample remains distinct from its other permutation variants, thus resolving denoising ambiguity; and 2) the reverse denoising process reconstructs polygonal shapes, initialized and directed by the guidance networks, as a conditional generation process subject to the sensor data. We have evaluated our approach for reconstructing two types of polygonal shapes: floorplan as a set of polygons and HD map for autonomous cars as a set of polylines. Through extensive experiments on standard benchmarks, we demonstrate that PolyDiffuse significantly advances the current state of the art and enables broader practical applications.