VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought

TL;DR

This paper introduces ICAL, a method enabling vision-language models to self-reflect and refine their experiences into high-quality, generalized strategies, significantly enhancing task performance with less human feedback.

Contribution

ICAL is a novel framework that allows VLM agents to abstract and improve their trajectories through self-reflection and iterative human feedback, leading to better decision-making and reduced manual effort.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Improves task success rates significantly over raw demonstrations.

Scales more efficiently, requiring less human feedback.

Abstract

Large-scale generative language and vision-language models (LLMs and VLMs) excel in few-shot learning but require high-quality demonstrations. We propose In-Context Abstraction Learning (ICAL), enabling VLM agents to transform suboptimal trajectories into high-quality training data through self-reflection and human feedback. Given imperfect task demonstrations, a VLM abstracts trajectories into generalized strategies and action annotations by correcting inefficiencies and annotating cognitive abstractions: causal relationships, object state changes, temporal subgoals, and task-relevant visual elements. These annotations are iteratively refined through human feedback during execution in similar environments. The resulting examples significantly improve decision-making when used for retrieval-augmented generation or fine-tuning. As the agent's example library grows, it becomes more efficient at abstracting new examples, requiring less human feedback and fewer environment interactions. ICAL achieves state-of-the-art results across multiple benchmarks. In TEACh dialogue-based instruction following, combining fine-tuning and retrieval on ICAL examples outperforms raw human demonstrations and expert examples by 17.5% in goal-condition success. In VisualWebArena, retrieval-augmented GPT-4V with ICAL improves task success 1.6x, while fine-tuned Qwen2-VL achieves 2.8x improvement over the base model. In Ego4D action forecasting, we surpass few-shot GPT-4V and remain competitive with supervised models. Our approach scales 2x better than raw demonstrations and significantly reduces manual prompt engineering requirements.