Unifying Language Agent Algorithms with Graph-based Orchestration Engine for Reproducible Agent Research

TL;DR

This paper introduces AGORA, a modular, graph-based framework that standardizes and simplifies the development, evaluation, and comparison of language agents powered by large language models, promoting reproducibility and efficiency.

Contribution

The paper presents a novel, extensible framework with a graph-based workflow engine, reusable algorithms, and evaluation tools for language agent research.

Findings

Sophisticated reasoning improves agent capabilities but increases computational cost.

Chain-of-Thought reasoning offers robust performance with lower overhead.

Extensive experiments reveal strengths and limitations of various agent algorithms.

Abstract

Language agents powered by large language models (LLMs) have demonstrated remarkable capabilities in understanding, reasoning, and executing complex tasks. However, developing robust agents presents significant challenges: substantial engineering overhead, lack of standardized components, and insufficient evaluation frameworks for fair comparison. We introduce Agent Graph-based Orchestration for Reasoning and Assessment (AGORA), a flexible and extensible framework that addresses these challenges through three key contributions: (1) a modular architecture with a graph-based workflow engine, efficient memory management, and clean component abstraction; (2) a comprehensive suite of reusable agent algorithms implementing state-of-the-art reasoning approaches; and (3) a rigorous evaluation framework enabling systematic comparison across multiple dimensions. Through extensive experiments on mathematical reasoning and multimodal tasks, we evaluate various agent algorithms across different LLMs, revealing important insights about their relative strengths and applicability. Our results demonstrate that while sophisticated reasoning approaches can enhance agent capabilities, simpler methods like Chain-of-Thought often exhibit robust performance with significantly lower computational overhead. AGORA not only simplifies language agent development but also establishes a foundation for reproducible agent research through standardized evaluation protocols.