BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence

TL;DR

The paper introduces BBScore, a novel reference-free metric based on Brownian bridge theory, for assessing both local and global text coherence, demonstrating competitive performance and applicability in distinguishing human and AI-generated texts.

Contribution

It proposes a new coherence metric that captures overarching text coherence without relying on static embeddings or end-to-end training, enhancing evaluation of long texts.

Findings

Achieves performance comparable to state-of-the-art methods on artificial discrimination tasks.

Effectively differentiates between human-written and AI-generated texts in downstream applications.

Demonstrates potential for generalization across various large language model styles.

Abstract

Measuring the coherence of text is a vital aspect of evaluating the quality of written content. Recent advancements in neural coherence modeling have demonstrated their efficacy in capturing entity coreference and discourse relations, thereby enhancing coherence evaluation. However, many existing methods heavily depend on static embeddings or focus narrowly on nearby context, constraining their capacity to measure the overarching coherence of long texts. In this paper, we posit that coherent texts inherently manifest a sequential and cohesive interplay among sentences, effectively conveying the central theme, purpose, or standpoint. To explore this abstract relationship, we introduce the "BBScore," a novel reference-free metric grounded in Brownian bridge theory for assessing text coherence. Our findings showcase that when synergized with a simple additional classification component, this metric attains a performance level comparable to state-of-the-art techniques on standard artificial discrimination tasks. We also establish in downstream tasks that this metric effectively differentiates between human-written documents and text generated by large language models under a specific domain. Furthermore, we illustrate the efficacy of this approach in detecting written styles attributed to diverse large language models, underscoring its potential for generalizability. In summary, we present a novel Brownian bridge coherence metric capable of measuring both local and global text coherence, while circumventing the need for end-to-end model training. This flexibility allows for its application in various downstream tasks.