PublicAgent: Multi-Agent Design Principles From an LLM-Based Open Data Analysis Framework

TL;DR

PublicAgent is a multi-agent framework leveraging specialized LLM agents to improve open data analysis workflows, addressing limitations of end-to-end models and enabling non-experts to access and analyze public datasets effectively.

Contribution

The paper introduces PublicAgent, a novel multi-agent architecture that decomposes complex data analysis tasks into specialized agents, demonstrating improved robustness and effectiveness across models and queries.

Findings

Specialization improves effectiveness regardless of model strength.

Universal and conditional agents have distinct effectiveness patterns.

Removing key agents causes significant failures.

Abstract

Open data repositories hold potential for evidence-based decision-making, yet are inaccessible to non-experts lacking expertise in dataset discovery, schema mapping, and statistical analysis. Large language models show promise for individual tasks, but end-to-end analytical workflows expose fundamental limitations: attention dilutes across growing contexts, specialized reasoning patterns interfere, and errors propagate undetected. We present PublicAgent, a multi-agent framework that addresses these limitations through decomposition into specialized agents for intent clarification, dataset discovery, analysis, and reporting. This architecture maintains focused attention within agent contexts and enables validation at each stage. Evaluation across five models and 50 queries derives five design principles for multi-agent LLM systems. First, specialization provides value independent of model strength--even the strongest model shows 97.5% agent win rates, with benefits orthogonal to model scale. Second, agents divide into universal (discovery, analysis) and conditional (report, intent) categories. Universal agents show consistent effectiveness (std dev 12.4%) while conditional agents vary by model (std dev 20.5%). Third, agents mitigate distinct failure modes--removing discovery or analysis causes catastrophic failures (243-280 instances), while removing report or intent causes quality degradation. Fourth, architectural benefits persist across task complexity with stable win rates (86-92% analysis, 84-94% discovery), indicating workflow management value rather than reasoning enhancement. Fifth, wide variance in agent effectiveness across models (42-96% for analysis) requires model-aware architecture design. These principles guide when and why specialization is necessary for complex analytical workflows while enabling broader access to public data through natural language interfaces.