Versatile Neural Processes for Learning Implicit Neural Representations

TL;DR

This paper introduces Versatile Neural Processes (VNP), an advanced framework that significantly improves the modeling of complex signals in implicit neural representations by using hierarchical global latent variables and an efficient encoding scheme.

Contribution

VNP enhances neural process architectures with a bottleneck encoder and hierarchical latent variables, enabling better approximation of complex signals in INRs.

Findings

VNP achieves superior modeling of 1D, 2D, and 3D signals.

VNP accurately learns 3D scene representations without fine-tuning.

The framework reduces computational costs while maintaining high modeling capacity.

Abstract

Representing a signal as a continuous function parameterized by neural network (a.k.a. Implicit Neural Representations, INRs) has attracted increasing attention in recent years. Neural Processes (NPs), which model the distributions over functions conditioned on partial observations (context set), provide a practical solution for fast inference of continuous functions. However, existing NP architectures suffer from inferior modeling capability for complex signals. In this paper, we propose an efficient NP framework dubbed Versatile Neural Processes (VNP), which largely increases the capability of approximating functions. Specifically, we introduce a bottleneck encoder that produces fewer and informative context tokens, relieving the high computational cost while providing high modeling capability. At the decoder side, we hierarchically learn multiple global latent variables that jointly model the global structure and the uncertainty of a function, enabling our model to capture the distribution of complex signals. We demonstrate the effectiveness of the proposed VNP on a variety of tasks involving 1D, 2D and 3D signals. Particularly, our method shows promise in learning accurate INRs w.r.t. a 3D scene without further finetuning. Code is available at https://github.com/ZongyuGuo/Versatile-NP .