ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering

TL;DR

This paper introduces ProxAnn, a human-centered evaluation protocol for topic models and document clustering that uses crowdworkers or LLMs to assess model outputs in a way that aligns with real-world usage, validated by extensive annotations.

Contribution

ProxAnn provides a scalable, human-aligned evaluation method for topic models and clustering, validated by LLM-based proxies matching human judgments.

Findings

LLM proxies are statistically indistinguishable from human annotators.

The protocol effectively captures real-world model usage.

Automated proxies can replace human evaluation in large-scale assessments.

Abstract

Topic model and document-clustering evaluations either use automated metrics that align poorly with human preferences or require expert labels that are intractable to scale. We design a scalable human evaluation protocol and a corresponding automated approximation that reflect practitioners' real-world usage of models. Annotators -- or an LLM-based proxy -- review text items assigned to a topic or cluster, infer a category for the group, then apply that category to other documents. Using this protocol, we collect extensive crowdworker annotations of outputs from a diverse set of topic models on two datasets. We then use these annotations to validate automated proxies, finding that the best LLM proxies are statistically indistinguishable from a human annotator and can therefore serve as a reasonable substitute in automated evaluations. Package, web interface, and data are at https://github.com/ahoho/proxann