PEER: Expertizing Domain-Specific Tasks with a Multi-Agent Framework and Tuning Methods

TL;DR

This paper introduces PEER, a multi-agent framework for domain-specific tasks that balances performance, cost, and data privacy, demonstrating near-GPT-4 performance in financial question-answering.

Contribution

The paper presents a novel multi-agent system with tuning strategies that improve domain-specific task performance while addressing cost and privacy concerns.

Findings

Achieves 95% of GPT-4's performance in financial QA

Effectively manages costs and data privacy

Provides best practice guidelines for multi-agent systems

Abstract

In domain-specific applications, GPT-4, augmented with precise prompts or Retrieval-Augmented Generation (RAG), shows notable potential but faces the critical tri-lemma of performance, cost, and data privacy. High performance requires sophisticated processing techniques, yet managing multiple agents within a complex workflow often proves costly and challenging. To address this, we introduce the PEER (Plan, Execute, Express, Review) multi-agent framework. This systematizes domain-specific tasks by integrating precise question decomposition, advanced information retrieval, comprehensive summarization, and rigorous self-assessment. Given the concerns of cost and data privacy, enterprises are shifting from proprietary models like GPT-4 to custom models, striking a balance between cost, security, and performance. We developed industrial practices leveraging online data and user feedback for efficient model tuning. This study provides best practice guidelines for applying multi-agent systems in domain-specific problem-solving and implementing effective agent tuning strategies. Our empirical studies, particularly in the financial question-answering domain, demonstrate that our approach achieves 95.0% of GPT-4's performance, while effectively managing costs and ensuring data privacy.