Attention Beats Linear for Fast Implicit Neural Representation Generation

TL;DR

This paper introduces an Attention-based Localized Implicit Neural Representation (ANR) that improves the efficiency and accuracy of modeling discontinuous signals, outperforming traditional MLP-based INRs in super-resolution tasks.

Contribution

The paper proposes a novel attention-based localized INR framework with a transformer-like hyper-network for efficient and accurate data representation and reconstruction.

Findings

Enhanced PSNR from 37.95dB to 47.25dB on CelebA dataset

Effective modeling of discontinuous signals with attention mechanisms

Improved super-resolution inference quality

Abstract

Implicit Neural Representation (INR) has gained increasing popularity as a data representation method, serving as a prerequisite for innovative generation models. Unlike gradient-based methods, which exhibit lower efficiency in inference, the adoption of hyper-network for generating parameters in Multi-Layer Perceptrons (MLP), responsible for executing INR functions, has surfaced as a promising and efficient alternative. However, as a global continuous function, MLP is challenging in modeling highly discontinuous signals, resulting in slow convergence during the training phase and inaccurate reconstruction performance. Moreover, MLP requires massive representation parameters, which implies inefficiencies in data representation. In this paper, we propose a novel Attention-based Localized INR (ANR) composed of a localized attention layer (LAL) and a global MLP that integrates coordinate features with data features and converts them to meaningful outputs. Subsequently, we design an instance representation framework that delivers a transformer-like hyper-network to represent data instances as a compact representation vector. With instance-specific representation vector and instance-agnostic ANR parameters, the target signals are well reconstructed as a continuous function. We further address aliasing artifacts with variational coordinates when obtaining the super-resolution inference results. Extensive experimentation across four datasets showcases the notable efficacy of our ANR method, e.g. enhancing the PSNR value from 37.95dB to 47.25dB on the CelebA dataset. Code is released at https://github.com/Roninton/ANR.