# Human-anchored longitudinal comparison of generative AI with a bias-calibrated LLM-as-judge

## TL;DR

This study tracks changes in three major AI models over time using human and AI judges, revealing how their performance and behavior evolve.

## Contribution

A novel human-anchored longitudinal evaluation framework with bias-calibrated LLM-as-judge for tracking service drift in evolving AI models.

## Key findings

- Three models showed divergent stability: one stable, one improving, and one degrading mid-study.
- LLM-as-judge calibration increased agreement with human raters (τ = 0.59–0.68) and reduced volatility.
- Safety metrics co-varied with drift events, indicating behavioral shifts rather than causal changes.

## Abstract

Service LLMs evolve without public changelogs, complicating reproducible evaluation. We present a preregistered human-anchored longitudinal study that tracks three major model families over ten weekly waves using a fixed prompt bank (N = 240) across six domains. Blinded human raters provided correctness judgments, and a bias-calibrated LLM-as-judge produced secondary pairwise preferences corrected weekly via a Bradley–Terry model. Mixed-effects modeling and change-point detection (PELT with MBIC penalty) identified significant service drift patterns. Results show divergent stability trajectories among models: one stable, one improving, and one degrading mid-study. Judge calibration increased agreement with humans (τ = 0.59–0.68) while reducing volatility. Safety metrics co-varied with drift events, suggesting behavioral shifts rather than confirmed causal changes. All data, prompts, rubrics, and parameter configurations are provided in supporting files S1–S6.