2D Representation for Unguided Single-View 3D Super-Resolution in Real-Time

TL;DR

This paper presents 2Dto3D-SR, a real-time 3D super-resolution framework that encodes 3D data into 2D representations, enabling the use of 2D super-resolution techniques without high-res RGB guidance.

Contribution

It introduces a novel 2D representation for 3D super-resolution that simplifies the process and supports lightweight, fast models suitable for real-time applications.

Findings

Swin Transformer model achieves state-of-the-art accuracy.

Vision Mamba model provides real-time performance with competitive results.

The framework effectively operates without high-resolution RGB data.

Abstract

We introduce 2Dto3D-SR, a versatile framework for real-time single-view 3D super-resolution that eliminates the need for high-resolution RGB guidance. Our framework encodes 3D data from a single viewpoint into a structured 2D representation, enabling the direct application of existing 2D image super-resolution architectures. We utilize the Projected Normalized Coordinate Code (PNCC) to represent 3D geometry from a visible surface as a regular image, thereby circumventing the complexities of 3D point-based or RGB-guided methods. This design supports lightweight and fast models adaptable to various deployment environments. We evaluate 2Dto3D-SR with two implementations: one using Swin Transformers for high accuracy, and another using Vision Mamba for high efficiency. Experiments show the Swin Transformer model achieves state-of-the-art accuracy on standard benchmarks, while the Vision Mamba model delivers competitive results at real-time speeds. This establishes our geometry-guided pipeline as a surprisingly simple yet viable and practical solution for real-world scenarios, especially where high-resolution RGB data is inaccessible.