Improving Interactive In-Context Learning from Natural Language Feedback

TL;DR

This paper introduces a training framework that enhances large language models' ability to learn interactively from natural language feedback, significantly improving their reasoning and adaptability across various tasks.

Contribution

It presents a scalable method to train models for multi-turn interactive learning from feedback, enabling better adaptation and transfer across domains.

Findings

Smaller models nearly match larger models in interactive learning performance.

Models trained with this method generalize well to out-of-distribution tasks.

Enhanced in-context plasticity improves models' ability to self-correct and adapt.

Abstract

Adapting one's thought process based on corrective feedback is an essential ability in human learning, particularly in collaborative settings. In contrast, the current large language model training paradigm relies heavily on modeling vast, static corpora. While effective for knowledge acquisition, it overlooks the interactive feedback loops essential for models to adapt dynamically to their context. In this work, we propose a framework that treats this interactive in-context learning ability not as an emergent property, but as a distinct, trainable skill. We introduce a scalable method that transforms single-turn verifiable tasks into multi-turn didactic interactions driven by information asymmetry. We first show that current flagship models struggle to integrate corrective feedback on hard reasoning tasks. We then demonstrate that models trained with our approach dramatically improve the ability to interactively learn from language feedback. More specifically, the multi-turn performance of a smaller model nearly reaches that of a model an order of magnitude larger. We also observe robust out-of-distribution generalization: interactive training on math problems transfers to diverse domains like coding, puzzles and maze navigation. Our qualitative analysis suggests that this improvement is due to an enhanced in-context plasticity. Finally, we show that this paradigm offers a unified path to self-improvement. By training the model to predict the teacher's critiques, effectively modeling the feedback environment, we convert this external signal into an internal capability, allowing the model to self-correct even without a teacher.