Robust Neural Processes for Noisy Data

TL;DR

This paper investigates the robustness of Neural Processes in noisy data scenarios, revealing that attention-based models are more affected by noise and proposing a training method to improve their robustness, with experiments showing superior performance across noise levels.

Contribution

The paper introduces a simple training method to enhance Neural Processes' robustness to noisy data, addressing the vulnerability of attention-based models to noise.

Findings

Attention-based models are more affected by noise, leading to overfitting.

The proposed training method improves robustness across all noise levels.

Models trained with the new method outperform existing NP models on various datasets.

Abstract

Models that adapt their predictions based on some given contexts, also known as in-context learning, have become ubiquitous in recent years. We propose to study the behavior of such models when data is contaminated by noise. Towards this goal we use the Neural Processes (NP) framework, as a simple and rigorous way to learn a distribution over functions, where predictions are based on a set of context points. Using this framework, we find that the models that perform best on clean data, are different than the models that perform best on noisy data. Specifically, models that process the context using attention, are more severely affected by noise, leading to in-context overfitting. We propose a simple method to train NP models that makes them more robust to noisy data. Experiments on 1D functions and 2D image datasets demonstrate that our method leads to models that outperform all other NP models for all noise levels.