Model Proficiency in Centralized Multi-Agent Systems: A Performance Study

TL;DR

This paper introduces a framework for assessing team proficiency in centralized multi-agent systems using three metrics, with simulation results showing effective real-time performance evaluation.

Contribution

It extends proficiency self-assessment from single agents to teams, proposing three metrics for centralized assessment and validating their effectiveness through simulations.

Findings

MPB and KS metrics accurately detect model mismatches

Metrics align well with KL divergence as a reference

Effective real-time proficiency assessment demonstrated

Abstract

Autonomous agents are increasingly deployed in dynamic environments where their ability to perform a given task depends on both individual and team-level proficiency. While proficiency self-assessment (PSA) has been studied for single agents, its extension to a team of agents remains underexplored. This letter addresses this gap by presenting a framework for team PSA in centralized settings. We investigate three metrics for centralized team PSA: the measurement prediction bound (MPB), the Kolmogorov-Smirnov (KS) statistic, and the Kullback-Leibler (KL) divergence. These metrics quantify the discrepancy between predicted and actual measurements. We use the KL divergence as a reference metric since it compares the true and predictive distributions, whereas the MPB and KS provide efficient indicators for in situ assessment. Simulation results in a target tracking scenario demonstrate that both MPB and KS metrics accurately capture model mismatches, align with the KL divergence reference, and enable real-time proficiency assessment.