Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomes

TL;DR

This paper investigates how different alignment techniques influence the effectiveness of LLMs as collaborative partners in multi-turn, multi-party interactions, highlighting the importance of robustness in intervention strategies for improved coordination.

Contribution

It introduces a novel roleplay simulation methodology and a theoretical framework to evaluate the impact of alignment methods on multi-agent collaboration outcomes.

Findings

Robust intervention agents outperform baseline alignments in task success.

Alignment methods need to consider multi-turn, multi-party dynamics.

Theoretical analysis reveals limitations of existing alignment assumptions.

Abstract

As Large Language Models (LLMs) get integrated into diverse workflows, they are increasingly being regarded as "collaborators" with humans, and required to work in coordination with other AI systems. If such AI collaborators are to reliably coordinate their actions and behaviors with humans or other AIs, their properties and behaviors over multi-turn interactions must be known and predictable. This paper examines how different alignment methods affect LLM agents' effectiveness as partners in multi-turn, multi-party collaborations. We study this question through the lens of intervention agents that insert themselves into group dialogues not to provide answers, but to encourage the collaborative group to slow down and reflect upon their reasoning for deliberative decision-making. Common alignment techniques are typically developed under simplified single-user settings and assume the optimality of the underlying token MDP. Using the theoretical lens of the modified-action MDP, we show how they do not account for the dynamics of long-horizon multi-party interactions. We present a novel roleplay simulation methodology, where we align LLMs according to different methods and then deploy them in collaborative task dialogues to quantify how interventions affect the trajectory of group collaboration, belief alignment, and coordination. Our results show that an intervention agent that is robust to action modification significantly outperforms common alignment baselines in supporting correct task outcomes.