Requesting Expert Reasoning: Augmenting LLM Agents with Learned Collaborative Intervention

TL;DR

This paper introduces AHCE, a framework that enhances LLM agents by learning when and how to request expert reasoning, significantly improving success rates in complex tasks with minimal human input.

Contribution

The paper presents a novel framework for on-demand human-AI collaboration that learns to effectively request expert reasoning, improving LLM agent performance in specialized domains.

Findings

Task success rates increased by 32% on normal difficulty tasks.

Nearly 70% success rate on highly difficult tasks.

Minimal human intervention required for effective collaboration.

Abstract

Large Language Model (LLM) based agents excel at general reasoning but often fail in specialized domains where success hinges on long-tail knowledge absent from their training data. While human experts can provide this missing knowledge, their guidance is often unstructured and unreliable, making its direct integration into an agent's plan problematic. To address this, we introduce AHCE (Active Human-Augmented Challenge Engagement), a framework for on-demand Human-AI collaboration. At its core, the Human Feedback Module (HFM) employs a learned policy to treat the human expert as an interactive reasoning tool. Extensive experiments in Minecraft demonstrate the framework's effectiveness, increasing task success rates by 32% on normal difficulty tasks and nearly 70% on highly difficult tasks, all with minimal human intervention. Our work demonstrates that successfully augmenting agents requires learning how to request expert reasoning, moving beyond simple requests for help.