Sequential Neural Processes

TL;DR

Sequential Neural Processes extend Neural Processes by incorporating temporal dynamics, enabling modeling of dynamic stochastic processes and 4D scene understanding, with demonstrated effectiveness in dynamic regression and scene rendering.

Contribution

The paper introduces Sequential Neural Processes, a novel extension of Neural Processes that models temporal dependencies in stochastic processes, including the first 4D model for dynamic 3D scene analysis.

Findings

Effective in dynamic non-stationary regression

Successful 4D scene inference and rendering

First model to handle temporal dynamics of 3D scenes

Abstract

Neural Processes combine the strengths of neural networks and Gaussian processes to achieve both flexible learning and fast prediction in stochastic processes. However, a large class of problems comprises underlying temporal dependency structures in a sequence of stochastic processes that Neural Processes (NP) do not explicitly consider. In this paper, we propose Sequential Neural Processes (SNP) which incorporates a temporal state-transition model of stochastic processes and thus extends its modeling capabilities to dynamic stochastic processes. In applying SNP to dynamic 3D scene modeling, we introduce the Temporal Generative Query Networks. To our knowledge, this is the first 4D model that can deal with the temporal dynamics of 3D scenes. In experiments, we evaluate the proposed methods in dynamic (non-stationary) regression and 4D scene inference and rendering.