Agent Identity Evals: Measuring Agentic Identity

TL;DR

This paper introduces a new empirical framework called agent identity evals (AIE) to measure and ensure the stability, reliability, and consistency of language model agents' identities over time, addressing key challenges inherited from large language models.

Contribution

The paper presents a novel, statistically-driven evaluation framework with metrics for assessing and maintaining agentic identity in language model agents, including methods applicable throughout their lifecycle.

Findings

AIE provides measurable metrics for agent identity stability.

AIE can be integrated with performance and robustness measures.

Worked examples demonstrate application of AIE methods.

Abstract

Central to agentic capability and trustworthiness of language model agents (LMAs) is the extent they maintain stable, reliable, identity over time. However, LMAs inherit pathologies from large language models (LLMs) (statelessness, stochasticity, sensitivity to prompts and linguistically-intermediation) which can undermine their identifiability, continuity, persistence and consistency. This attrition of identity can erode their reliability, trustworthiness and utility by interfering with their agentic capabilities such as reasoning, planning and action. To address these challenges, we introduce \textit{agent identity evals} (AIE), a rigorous, statistically-driven, empirical framework for measuring the degree to which an LMA system exhibit and maintain their agentic identity over time, including their capabilities, properties and ability to recover from state perturbations. AIE comprises a set of novel metrics which can integrate with other measures of performance, capability and agentic robustness to assist in the design of optimal LMA infrastructure and scaffolding such as memory and tools. We set out formal definitions and methods that can be applied at each stage of the LMA life-cycle, and worked examples of how to apply them.