Contrastive Conditional Neural Processes

TL;DR

This paper introduces contrastive learning techniques into Conditional Neural Processes to improve function representation and prediction, especially in high-dimensional noisy settings, by decoupling generative and meta-learning objectives.

Contribution

It proposes hierarchical contrastive branches (TCL and FCL) to enhance CNPs, enabling better local and global function understanding, and demonstrates improved performance across various data dimensions.

Findings

TCL captures high-level observation abstractions.

FCL improves function identification.

Model outperforms existing CNP variants in diverse settings.

Abstract

Conditional Neural Processes~(CNPs) bridge neural networks with probabilistic inference to approximate functions of Stochastic Processes under meta-learning settings. Given a batch of non-{\it i.i.d} function instantiations, CNPs are jointly optimized for in-instantiation observation prediction and cross-instantiation meta-representation adaptation within a generative reconstruction pipeline. There can be a challenge in tying together such two targets when the distribution of function observations scales to high-dimensional and noisy spaces. Instead, noise contrastive estimation might be able to provide more robust representations by learning distributional matching objectives to combat such inherent limitation of generative models. In light of this, we propose to equip CNPs by 1) aligning prediction with encoded ground-truth observation, and 2) decoupling meta-representation adaptation from generative reconstruction. Specifically, two auxiliary contrastive branches are set up hierarchically, namely in-instantiation temporal contrastive learning~({\tt TCL}) and cross-instantiation function contrastive learning~({\tt FCL}), to facilitate local predictive alignment and global function consistency, respectively. We empirically show that {\tt TCL} captures high-level abstraction of observations, whereas {\tt FCL} helps identify underlying functions, which in turn provides more efficient representations. Our model outperforms other CNPs variants when evaluating function distribution reconstruction and parameter identification across 1D, 2D and high-dimensional time-series.