OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence

TL;DR

OpenHospital is an interactive platform designed to evolve and benchmark LLM-based collective intelligence in medical scenarios, enabling rapid capability enhancement and robust evaluation of agent systems.

Contribution

It introduces a novel arena for evolving and benchmarking LLM-based collective intelligence specifically in medical contexts, filling a key gap in current research.

Findings

Effective in fostering collective intelligence among agents

Provides robust metrics for benchmarking medical proficiency

Demonstrates significant capability improvements in experiments

Abstract

Large Language Model (LLM)-based Collective Intelligence (CI) presents a promising approach to overcoming the data wall and continuously boosting the capabilities of LLM agents. However, there is currently no dedicated arena for evolving and benchmarking LLM-based CI. To address this gap, we introduce OpenHospital, an interactive arena where physician agents can evolve CI through interactions with patient agents. This arena employs a data-in-agent-self paradigm that rapidly enhances agent capabilities and provides robust evaluation metrics for benchmarking both medical proficiency and system efficiency. Experiments demonstrate the effectiveness of OpenHospital in both fostering and quantifying CI.