A Flexible Multi-Agent LLM-Human Framework for Fast Human Validated Tool Building

TL;DR

This paper presents CollabToolBuilder, a multi-agent LLM framework with human-in-the-loop guidance that efficiently creates and validates tools aligned with human goals, reducing adaptation time and feedback effort.

Contribution

It introduces a novel multi-agent system integrating in-context learning and HITL controls for rapid, human-aligned tool development in complex iterative tasks.

Findings

Preliminary experiments demonstrate state-of-the-art research paper generation.

The framework effectively incorporates human feedback to refine tool creation.

Applicable to various iterative scientific and technical problem-solving tasks.

Abstract

We introduce CollabToolBuilder, a flexible multiagent LLM framework with expert-in-the-loop (HITL) guidance that iteratively learns to create tools for a target goal, aligning with human intent and process, while minimizing time for task/domain adaptation effort and human feedback capture. The architecture generates and validates tools via four specialized agents (Coach, Coder, Critic, Capitalizer) using a reinforced dynamic prompt and systematic human feedback integration to reinforce each agent's role toward goals and constraints. This work is best viewed as a system-level integration and methodology combining multi-agent in-context learning, HITL controls, and reusable tool capitalization for complex iterative problems such as scientific document generation. We illustrate it with preliminary experiments (e.g., generating state-of-the-art research papers or patents given an abstract) and discuss its applicability to other iterative problem-solving.