Locality-Aware Generalizable Implicit Neural Representation

TL;DR

This paper introduces a locality-aware generalizable implicit neural representation framework that combines transformers and multi-band modulation to better capture fine details and improve performance in tasks like image generation.

Contribution

It proposes a novel framework integrating a transformer encoder with a locality-aware INR decoder for enhanced local detail encoding in generalizable INRs.

Findings

Outperforms previous generalizable INRs significantly.

Effectively captures local spatial and spectral information.

Improves downstream tasks such as image generation.

Abstract

Generalizable implicit neural representation (INR) enables a single continuous function, i.e., a coordinate-based neural network, to represent multiple data instances by modulating its weights or intermediate features using latent codes. However, the expressive power of the state-of-the-art modulation is limited due to its inability to localize and capture fine-grained details of data entities such as specific pixels and rays. To address this issue, we propose a novel framework for generalizable INR that combines a transformer encoder with a locality-aware INR decoder. The transformer encoder predicts a set of latent tokens from a data instance to encode local information into each latent token. The locality-aware INR decoder extracts a modulation vector by selectively aggregating the latent tokens via cross-attention for a coordinate input and then predicts the output by progressively decoding with coarse-to-fine modulation through multiple frequency bandwidths. The selective token aggregation and the multi-band feature modulation enable us to learn locality-aware representation in spatial and spectral aspects, respectively. Our framework significantly outperforms previous generalizable INRs and validates the usefulness of the locality-aware latents for downstream tasks such as image generation.