HEAT: Holistic Edge Attention Transformer for Structured Reconstruction

TL;DR

HEAT introduces a comprehensive attention-based neural network that reconstructs geometric structures from 2D images by detecting corners and classifying edges, outperforming existing methods in structured reconstruction tasks.

Contribution

The paper proposes a holistic edge classification architecture with innovative feature initialization, image feature fusion, and a masked training strategy for structured reconstruction.

Findings

Outperforms state-of-the-art in outdoor building and indoor floorplan reconstruction

Effective end-to-end corner detection and edge classification

Demonstrates robustness across different structured reconstruction tasks

Abstract

This paper presents a novel attention-based neural network for structured reconstruction, which takes a 2D raster image as an input and reconstructs a planar graph depicting an underlying geometric structure. The approach detects corners and classifies edge candidates between corners in an end-to-end manner. Our contribution is a holistic edge classification architecture, which 1) initializes the feature of an edge candidate by a trigonometric positional encoding of its end-points; 2) fuses image feature to each edge candidate by deformable attention; 3) employs two weight-sharing Transformer decoders to learn holistic structural patterns over the graph edge candidates; and 4) is trained with a masked learning strategy. The corner detector is a variant of the edge classification architecture, adapted to operate on pixels as corner candidates. We conduct experiments on two structured reconstruction tasks: outdoor building architecture and indoor floorplan planar graph reconstruction. Extensive qualitative and quantitative evaluations demonstrate the superiority of our approach over the state of the art. Code and pre-trained models are available at https://heat-structured-reconstruction.github.io.