Robustifying Sequential Neural Processes

TL;DR

This paper introduces Recurrent Memory Reconstruction (RMR), a novel attention mechanism that enhances meta-transfer learning by effectively handling dynamic tasks with limited context, significantly improving Sequential Neural Processes (SNP).

Contribution

The paper proposes RMR, a new attention mechanism, and integrates it into SNP to create ASNP-RMR, addressing underfitting and ineffective attention issues in meta-transfer learning.

Findings

RMR improves attention effectiveness in meta-transfer learning.

ASNP-RMR outperforms baseline models across various tasks.

Standard attention mechanisms are less effective in dynamic meta-transfer settings.

Abstract

When tasks change over time, meta-transfer learning seeks to improve the efficiency of learning a new task via both meta-learning and transfer-learning. While the standard attention has been effective in a variety of settings, we question its effectiveness in improving meta-transfer learning since the tasks being learned are dynamic and the amount of context can be substantially smaller. In this paper, using a recently proposed meta-transfer learning model, Sequential Neural Processes (SNP), we first empirically show that it suffers from a similar underfitting problem observed in the functions inferred by Neural Processes. However, we further demonstrate that unlike the meta-learning setting, the standard attention mechanisms are not effective in meta-transfer setting. To resolve, we propose a new attention mechanism, Recurrent Memory Reconstruction (RMR), and demonstrate that providing an imaginary context that is recurrently updated and reconstructed with interaction is crucial in achieving effective attention for meta-transfer learning. Furthermore, incorporating RMR into SNP, we propose Attentive Sequential Neural Processes-RMR (ASNP-RMR) and demonstrate in various tasks that ASNP-RMR significantly outperforms the baselines.