ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents

TL;DR

ReflecTool is a reflection-aware framework that enhances clinical agents by utilizing domain-specific tools and long-term memory, significantly improving performance across diverse medical tasks compared to existing methods.

Contribution

The paper introduces ReflecTool, a novel framework that leverages long-term memory and reflection to improve tool usage in clinical agents, along with a comprehensive benchmark for evaluation.

Findings

ReflecTool outperforms pure LLMs by over 10 points on the ClinicalAgent Benchmark.

ReflecTool surpasses existing agent-based methods by 3 points.

Extensive experiments demonstrate its adaptability and effectiveness in complex clinical tasks.

Abstract

Large Language Models (LLMs) have shown promising potential in the medical domain, assisting with tasks like clinical note generation and patient communication. However, current LLMs are limited to text-based communication, hindering their ability to interact with diverse forms of information in clinical environments. Despite clinical agents succeeding in diverse signal interaction, they are oriented to a single clinical scenario and hence fail for broader applications. To evaluate clinical agents holistically, we propose ClinicalAgent Bench~(CAB), a comprehensive medical agent benchmark consisting of 18 tasks across five key realistic clinical dimensions. Building on this, we introduce ReflecTool, a novel framework that excels at utilizing domain-specific tools within two stages. The first optimization stage progressively enlarges a long-term memory by saving successful solving processes and tool-wise experience of agents in a tiny pre-defined training set. In the following inference stage, ReflecTool can search for supportive successful demonstrations from already built long-term memory to guide the tool selection strategy, and a verifier improves the tool usage according to the tool-wise experience with two verification methods--iterative refinement and candidate selection. Extensive experiments on ClinicalAgent Benchmark demonstrate that ReflecTool surpasses the pure LLMs with more than 10 points and the well-established agent-based methods with 3 points, highlighting its adaptability and effectiveness in solving complex clinical tasks.