The Critical Role of Aspects in Measuring Document Similarity

TL;DR

This paper introduces ASPECTSIM, a framework for measuring document similarity based on explicit aspects, demonstrating its superiority over holistic methods and highlighting the importance of aspect conditioning in similarity assessments.

Contribution

The paper presents ASPECTSIM, a novel, interpretable framework for aspect-conditioned document similarity, and provides extensive evaluation showing its effectiveness and limitations across different models.

Findings

ASPECTSIM with GPT-4o achieves ~80% higher human-machine agreement than holistic methods.

Two-stage refinement significantly improves open-source LLMs' agreement with human judgments.

Large proprietary models outperform smaller open-source LLMs in aspect-conditioned similarity tasks.

Abstract

We introduce ASPECTSIM, a simple and interpretable framework that requires conditioning document similarity on an explicitly specified aspect, which is different from the traditional holistic approach in measuring document similarity. Experimenting with a newly constructed benchmark of 26K aspect-document pairs, we found that ASPECTSIM, when implemented with direct GPT-4o prompting, achieves substantially higher human-machine agreement ($\approx$80% higher) than the same for holistic similarity without explicit aspects. These findings underscore the importance of explicitly accounting for aspects when measuring document similarity and highlight the need to revise standard practice. Next, we conducted a large-scale meta-evaluation using 16 smaller open-source LLMs and 9 embedding models with a focus on making ASPECTSIM accessible and reproducible. While directly prompting LLMs to produce ASPECTSIM scores turned out be ineffective (20-30% human-machine agreement), a simple two-stage refinement improved their agreement by $\approx$140%. Nevertheless, agreement remains well below that of GPT-4o-based models, indicating that smaller open-source LLMs still lag behind large proprietary models in capturing aspect-conditioned similarity.