Tell Me What To Learn: Generalizing Neural Memory to be Controllable in Natural Language

TL;DR

This paper introduces a generalized neural memory system that allows natural language instructions to control what information is learned or ignored, enabling more flexible and adaptive continual learning in dynamic environments.

Contribution

It proposes a novel neural memory framework that incorporates natural language instructions for selective and controllable memory updates, addressing limitations of fixed-objective memory models.

Findings

Supports heterogeneous information sources effectively

Enables flexible, instruction-driven memory updates

Improves adaptability in real-world applications

Abstract

Modern machine learning models are deployed in diverse, non-stationary environments where they must continually adapt to new tasks and evolving knowledge. Continual fine-tuning and in-context learning are costly and brittle, whereas neural memory methods promise lightweight updates with minimal forgetting. However, existing neural memory models typically assume a single fixed objective and homogeneous information streams, leaving users with no control over what the model remembers or ignores over time. To address this challenge, we propose a generalized neural memory system that performs flexible updates based on learning instructions specified in natural language. Our approach enables adaptive agents to learn selectively from heterogeneous information sources, supporting settings, such as healthcare and customer service, where fixed-objective memory updates are insufficient.