CornerFormer: Boosting Corner Representation for Fine-Grained Structured Reconstruction

TL;DR

CornerFormer introduces an integrated approach that enhances corner representation by sharing features across detection and prediction tasks, significantly improving fine-grained structural reconstruction in images.

Contribution

It proposes a novel unified corner representation method that fuses knowledge between corner detection and edge prediction, outperforming existing two-stage transformer-based models.

Findings

Outperforms state-of-the-art by +1.9% F-1 on corners

Achieves +3.0% F-1 improvement on edges

Better reconstructs fine-grained structures like tiny edges

Abstract

Structured reconstruction is a non-trivial dense prediction problem, which extracts structural information (\eg, building corners and edges) from a raster image, then reconstructs it to a 2D planar graph accordingly. Compared with common segmentation or detection problems, it significantly relays on the capability that leveraging holistic geometric information for structural reasoning. Current transformer-based approaches tackle this challenging problem in a two-stage manner, which detect corners in the first model and classify the proposed edges (corner-pairs) in the second model. However, they separate two-stage into different models and only share the backbone encoder. Unlike the existing modeling strategies, we present an enhanced corner representation method: 1) It fuses knowledge between the corner detection and edge prediction by sharing feature in different granularity; 2) Corner candidates are proposed in four heatmap channels w.r.t its direction. Both qualitative and quantitative evaluations demonstrate that our proposed method can better reconstruct fine-grained structures, such as adjacent corners and tiny edges. Consequently, it outperforms the state-of-the-art model by +1.9\%@F-1 on Corner and +3.0\%@F-1 on Edge.