Cochain: Balancing Insufficient and Excessive Collaboration in LLM Agent Workflows

TL;DR

Cochain is a novel framework that enhances business workflow collaboration in LLMs by integrating knowledge graphs and prompt trees, reducing costs and outperforming existing methods in prompt engineering and multi-agent systems.

Contribution

Cochain introduces a new collaboration prompting framework combining knowledge graphs and prompt trees to improve efficiency and effectiveness in LLM-based business workflows.

Findings

Cochain outperforms all baselines in prompt engineering and multi-agent LLMs.

Expert evaluations show small models with Cochain outperform GPT-4.

Cochain reduces costs while maintaining high performance in complex tasks.

Abstract

Large Language Models (LLMs) have demonstrated impressive performance in executing complex reasoning tasks. Chain-of-thought effectively enhances reasoning capabilities by unlocking the potential of large models, while multi-agent systems provide more comprehensive solutions by integrating the collective intelligence of multiple agents. However, both approaches face significant limitations. Single-agent with chain-of-thought, due to the inherent complexity of designing cross-domain prompts, faces collaboration challenges. Meanwhile, multi-agent systems consume substantial tokens and inevitably dilute the primary problem, which is particularly problematic in business workflow tasks. To address these challenges, we propose Cochain, a collaboration prompting framework that effectively solves the business workflow collaboration problem by combining knowledge and prompts at a reduced cost. Specifically, we construct an integrated knowledge graph that incorporates knowledge from multiple stages. Furthermore, by maintaining and retrieving a prompts tree, we can obtain prompt information relevant to other stages of the business workflow. We perform extensive evaluations of Cochain across multiple datasets, demonstrating that Cochain outperforms all baselines in both prompt engineering and multi-agent LLMs. Additionally, expert evaluation results indicate that the use of a small model in combination with Cochain outperforms GPT-4.