RendNet: Unified 2D/3D Recognizer With Latent Space Rendering

TL;DR

RendNet is a unified recognition architecture that leverages both vector and raster graphics representations, utilizing a rendering process to improve 2D and 3D object recognition accuracy.

Contribution

It introduces a novel unified model that combines vector and raster graphics recognition through a differentiable rendering process, enhancing recognition performance.

Findings

Achieves state-of-the-art results on multiple VG datasets.

Effectively combines VG and RG information for improved recognition.

Demonstrates robustness across 2D and 3D recognition tasks.

Abstract

Vector graphics (VG) have been ubiquitous in our daily life with vast applications in engineering, architecture, designs, etc. The VG recognition process of most existing methods is to first render the VG into raster graphics (RG) and then conduct recognition based on RG formats. However, this procedure discards the structure of geometries and loses the high resolution of VG. Recently, another category of algorithms is proposed to recognize directly from the original VG format. But it is affected by the topological errors that can be filtered out by RG rendering. Instead of looking at one format, it is a good solution to utilize the formats of VG and RG together to avoid these shortcomings. Besides, we argue that the VG-to-RG rendering process is essential to effectively combine VG and RG information. By specifying the rules on how to transfer VG primitives to RG pixels, the rendering process depicts the interaction and correlation between VG and RG. As a result, we propose RendNet, a unified architecture for recognition on both 2D and 3D scenarios, which considers both VG/RG representations and exploits their interaction by incorporating the VG-to-RG rasterization process. Experiments show that RendNet can achieve state-of-the-art performance on 2D and 3D object recognition tasks on various VG datasets.