NovAScore: A New Automated Metric for Evaluating Document Level Novelty
Lin Ai, Ziwei Gong, Harshsaiprasad Deshpande, Alexander Johnson, Emmy, Phung, Ahmad Emami, Julia Hirschberg

TL;DR
NovAScore is an automated, interpretable metric designed to evaluate document-level novelty, addressing the limitations of human annotation and correlating well with human judgments.
Contribution
The paper introduces NovAScore, a novel automated metric for evaluating document novelty that combines atomic information scores with dynamic weighting for improved interpretability and flexibility.
Findings
Achieves 0.626 correlation with human judgments on TAP-DLND 1.0 dataset.
Achieves 0.920 Pearson correlation on internal human-annotated dataset.
Provides detailed analysis of a document's novelty and importance.
Abstract
The rapid expansion of online content has intensified the issue of information redundancy, underscoring the need for solutions that can identify genuinely new information. Despite this challenge, the research community has seen a decline in focus on novelty detection, particularly with the rise of large language models (LLMs). Additionally, previous approaches have relied heavily on human annotation, which is time-consuming, costly, and particularly challenging when annotators must compare a target document against a vast number of historical documents. In this work, we introduce NovAScore (Novelty Evaluation in Atomicity Score), an automated metric for evaluating document-level novelty. NovAScore aggregates the novelty and salience scores of atomic information, providing high interpretability and a detailed analysis of a document's novelty. With its dynamic weight adjustment scheme,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsScientific Computing and Data Management
MethodsFocus
