Detecting Perspective Shifts in Multi-agent Systems

TL;DR

This paper introduces TDKPS, a novel framework for monitoring behavioral changes in black-box multi-agent systems over time, enabling detection of significant shifts correlated with external events.

Contribution

The paper presents TDKPS, the first principled method for embedding and analyzing temporal behavioral dynamics in multi-agent systems.

Findings

TDKPS effectively detects behavioral changes in simulated multi-agent systems.

The proposed tests are sensitive to key hyperparameters and can identify significant shifts.

Application to real-world data shows TDKPS detects changes correlated with external events.

Abstract

Generative models augmented with external tools and update mechanisms (or \textit{agents}) have demonstrated capabilities beyond intelligent prompting of base models. As agent use proliferates, dynamic multi-agent systems have naturally emerged. Recent work has investigated the theoretical and empirical properties of low-dimensional representations of agents based on query responses at a single time point. This paper introduces the Temporal Data Kernel Perspective Space (TDKPS), which jointly embeds agents across time, and proposes several novel hypothesis tests for detecting behavioral change at the agent- and group-level in black-box multi-agent systems. We characterize the empirical properties of our proposed tests, including their sensitivity to key hyperparameters, in simulations motivated by a multi-agent system of evolving digital personas. Finally, we demonstrate via natural experiment that our proposed tests detect changes that correlate sensitively, specifically, and significantly with a real exogenous event. As far as we are aware, TDKPS is the first principled framework for monitoring behavioral dynamics in black-box multi-agent systems -- a critical capability as generative agent deployment continues to scale.