BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation

TL;DR

This paper introduces BBScoreV2, a new evaluation metric for language models that leverages stochastic latent representations to assess temporal and structural consistency in long text sequences.

Contribution

It presents a novel likelihood-based evaluation metric, BBScoreV2, that captures time-evolution and latent alignment in language model embeddings using stochastic processes.

Findings

Stochastic latent space induces a temporal ordering of model representations.

BBScoreV2 improves evaluation of temporal consistency and AI-generated content detection.

The method aligns with natural language properties and enhances interpretability.

Abstract

Autoregressive generative models play a key role in various language tasks, especially for modeling and evaluating long text sequences. While recent methods leverage stochastic representations to better capture sequence dynamics, encoding both temporal and structural dependencies and utilizing such information for evaluation remains challenging. In this work, we observe that fitting transformer-based model embeddings into a stochastic process yields ordered latent representations from originally unordered model outputs. Building on this insight and prior work, we theoretically introduce a novel likelihood-based evaluation metric BBScoreV2. Empirically, we demonstrate that the stochastic latent space induces a "clustered-to-temporal ordered" mapping of language model representations in high-dimensional space, offering both intuitive and quantitative support for the effectiveness of BBScoreV2. Furthermore, this structure aligns with intrinsic properties of natural language and enhances performance on tasks such as temporal consistency evaluation (e.g., Shuffle tasks) and AI-generated content detection.