Reevaluating Meta-Learning Optimization Algorithms Through Contextual Self-Modulation

TL;DR

This paper extends Contextual Self-Modulation (CSM) for broader applicability and scalability, demonstrating its effectiveness across diverse tasks and integrating it into other meta-learning frameworks, with insights on higher-order approximations.

Contribution

Introduces iCSM and StochasticNCF to enhance CSM's scalability and applicability, and integrates CSM into FlashCAVIA, advancing meta-learning techniques for physical and diverse systems.

Findings

iCSM enables infinite-dimensional context embedding.

StochasticNCF offers low-cost meta-gradient approximation.

Higher-order Taylor expansions do not necessarily improve generalization.

Abstract

Contextual Self-Modulation (CSM) (Nzoyem et al., 2025) is a potent regularization mechanism for Neural Context Flows (NCFs) which demonstrates powerful meta-learning on physical systems. However, CSM has limitations in its applicability across different modalities and in high-data regimes. In this work, we introduce two extensions: $i$CSM which expands CSM to infinite-dimensional variations by embedding the contexts into a function space, and StochasticNCF which improves scalability by providing a low-cost approximation of meta-gradient updates through a sampled set of nearest environments. These extensions are demonstrated through comprehensive experimentation on a range of tasks, including dynamical systems, computer vision challenges, and curve fitting problems. Additionally, we incorporate higher-order Taylor expansions via Taylor-Mode automatic differentiation, revealing that higher-order approximations do not necessarily enhance generalization. Finally, we demonstrate how CSM can be integrated into other meta-learning frameworks with FlashCAVIA, a computationally efficient extension of the CAVIA meta-learning framework (Zintgraf et al., 2019). Together, these contributions highlight the significant benefits of CSM and indicate that its strengths in meta-learning and out-of-distribution tasks are particularly well-suited to physical systems. Our open-source library, designed for modular integration of self-modulation into contextual meta-learning workflows, is available at https://github.com/ddrous/self-mod.