ChatCollab: Exploring Collaboration Between Humans and AI Agents in Software Teams

TL;DR

ChatCollab introduces a flexible framework for human-AI collaboration in software teams, enabling agents to autonomously coordinate roles and tasks within Slack, with promising results in software development tasks.

Contribution

We present a novel architecture for multi-agent collaboration that supports role flexibility and autonomous coordination between humans and AI in team environments.

Findings

AI agents successfully identify roles and responsibilities.

ChatCollab AI agents produce comparable or better software than prior systems.

Differentiated roles influence collaboration dynamics, e.g., AI CEO suggests more often.

Abstract

We explore the potential for productive team-based collaboration between humans and Artificial Intelligence (AI) by presenting and conducting initial tests with a general framework that enables multiple human and AI agents to work together as peers. ChatCollab's novel architecture allows agents - human or AI - to join collaborations in any role, autonomously engage in tasks and communication within Slack, and remain agnostic to whether their collaborators are human or AI. Using software engineering as a case study, we find that our AI agents successfully identify their roles and responsibilities, coordinate with other agents, and await requested inputs or deliverables before proceeding. In relation to three prior multi-agent AI systems for software development, we find ChatCollab AI agents produce comparable or better software in an interactive game development task. We also propose an automated method for analyzing collaboration dynamics that effectively identifies behavioral characteristics of agents with distinct roles, allowing us to quantitatively compare collaboration dynamics in a range of experimental conditions. For example, in comparing ChatCollab AI agents, we find that an AI CEO agent generally provides suggestions 2-4 times more often than an AI product manager or AI developer, suggesting agents within ChatCollab can meaningfully adopt differentiated collaborative roles. Our code and data can be found at: https://github.com/ChatCollab.