Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidance

TL;DR

This paper introduces ARIA, a framework enabling large language models to learn and adapt at test time through human-in-the-loop guidance, improving performance in dynamic environments.

Contribution

The paper presents ARIA, a novel self-improving agent framework that continuously updates its knowledge during operation using structured self-dialogue and human feedback.

Findings

ARIA outperforms baseline methods in adaptability and accuracy.

Effective knowledge updating reduces errors in dynamic tasks.

Deployed in TikTok Pay, ARIA handles over 150 million users effectively.

Abstract

Large language model (LLM) agents often struggle in environments where rules and required domain knowledge frequently change, such as regulatory compliance and user risk screening. Current approaches, like offline fine-tuning and standard prompting, are insufficient because they cannot effectively adapt to new knowledge during actual operation. To address this limitation, we propose the Adaptive Reflective Interactive Agent (ARIA), an LLM agent framework designed specifically to continuously learn updated domain knowledge at test time. ARIA assesses its own uncertainty through structured self-dialogue, proactively identifying knowledge gaps and requesting targeted explanations or corrections from human experts. It then systematically updates an internal, timestamped knowledge repository with provided human guidance, detecting and resolving conflicting or outdated knowledge through comparisons and clarification queries. We evaluate ARIA on the realistic customer due diligence name screening task on TikTok Pay, alongside publicly available dynamic knowledge tasks. Results demonstrate significant improvements in adaptability and accuracy compared to baselines using standard offline fine-tuning and existing self-improving agents. ARIA is deployed within TikTok Pay serving over 150 million monthly active users, confirming its practicality and effectiveness for operational use in rapidly evolving environments.