Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models

TL;DR

This paper introduces RICOL, a framework that uses large language models to improve temporal credit assignment in reinforcement learning, achieving high sample efficiency and comparable performance to traditional methods.

Contribution

It proposes a novel retrospective in-context learning approach leveraging LLMs for dense reward estimation, enhancing sample efficiency in RL.

Findings

RICL accurately estimates the advantage function with limited samples.

RICOL achieves comparable performance to traditional RL algorithms.

The approach improves sample efficiency in four BabyAI scenarios.

Abstract

Learning from self-sampled data and sparse environmental feedback remains a fundamental challenge in training self-evolving agents. Temporal credit assignment mitigates this issue by transforming sparse feedback into dense supervision signals. However, previous approaches typically depend on learning task-specific value functions for credit assignment, which suffer from poor sample efficiency and limited generalization. In this work, we propose to leverage pretrained knowledge from large language models (LLMs) to transform sparse rewards into dense training signals (i.e., the advantage function) through retrospective in-context learning (RICL). We further propose an online learning framework, RICOL, which iteratively refines the policy based on the credit assignment results from RICL. We empirically demonstrate that RICL can accurately estimate the advantage function with limited samples and effectively identify critical states in the environment for temporal credit assignment. Extended evaluation on four BabyAI scenarios show that RICOL achieves comparable convergent performance with traditional online RL algorithms with significantly higher sample efficiency. Our findings highlight the potential of leveraging LLMs for temporal credit assignment, paving the way for more sample-efficient and generalizable RL paradigms.