MetaReflection: Learning Instructions for Language Agents using Past Reflections

TL;DR

MetaReflection is an offline reinforcement learning method that improves language agents by leveraging past experiences to enhance performance across diverse complex tasks with fewer LLM calls.

Contribution

It introduces MetaReflection, a novel offline reinforcement learning approach that augments semantic memory for language agents, enabling performance improvements without online reflection.

Findings

Boosts agent performance by up to 16.82% over GPT-4 baseline

Performs comparably to state-of-the-art prompt optimization techniques

Requires fewer LLM calls for effective performance

Abstract

The popularity of Large Language Models (LLMs) have unleashed a new age ofLanguage Agents for solving a diverse range of tasks. While contemporary frontier LLMs are capable enough to power reasonably good Language agents, the closed-API model makes it hard to improve in cases they perform sub-optimally. To address this, recent works have explored ways to improve their performance using techniques like self-reflection and prompt optimization. Unfortunately, techniques like self-reflection can be used only in an online setup, while contemporary prompt optimization techniques are designed and tested to work on simple tasks. To this end, we introduce MetaReflection, a novel offline reinforcement learning technique that enhances the performance of Language Agents by augmenting a semantic memory based on experiential learnings from past trials. We demonstrate the efficacy of MetaReflection by evaluating across multiple domains, including complex logical reasoning, biomedical semantic similarity, open world question answering, and vulnerability threat detection, in Infrastructure-as-Code, spanning different agent designs. MetaReflection boosts Language agents' performance by 4% to 16.82% over the raw GPT-4 baseline and performs on par with existing state-of-the-art prompt optimization techniques while requiring fewer LLM calls.