Distilling Feedback into Memory-as-a-Tool

TL;DR

This paper introduces a framework that converts transient feedback into retrievable guidelines using a memory system, enabling large language models to perform reasoning more efficiently and cost-effectively.

Contribution

The paper presents a novel memory-based approach for integrating feedback into LLM reasoning, reducing inference costs compared to traditional test-time refinement methods.

Findings

Achieves comparable performance to test-time refinement with lower inference costs

Introduces the Rubric Feedback Bench dataset for rubric-based learning evaluation

Demonstrates rapid adaptation of augmented LLMs to feedback

Abstract

We propose a framework that amortizes the cost of inference-time reasoning by converting transient critiques into retrievable guidelines, through a file-based memory system and agent-controlled tool calls. We evaluate this method on the Rubric Feedback Bench, a novel dataset for rubric-based learning. Experiments demonstrate that our augmented LLMs rapidly match the performance of test-time refinement pipelines while drastically reducing inference cost.