NewsEdits 2.0: Learning the Intentions Behind Updating News

TL;DR

This paper introduces a new schema for classifying news article updates, demonstrates that linguistic features can predict factual updates with high precision, and applies this to improve language model abstention accuracy.

Contribution

It presents the NewsEdits 2.0 taxonomy, a method for predicting factual updates in news articles using only text, and shows its effectiveness in enhancing LLM abstention.

Findings

High-precision prediction of factual updates

Effective separation of update types in news revisions

Improved LLM abstention accuracy

Abstract

As events progress, news articles often update with new information: if we are not cautious, we risk propagating outdated facts. In this work, we hypothesize that linguistic features indicate factual fluidity, and that we can predict which facts in a news article will update using solely the text of a news article (i.e. not external resources like search engines). We test this hypothesis, first, by isolating fact-updates in large news revisions corpora. News articles may update for many reasons (e.g. factual, stylistic, narrative). We introduce the NewsEdits 2.0 taxonomy, an edit-intentions schema that separates fact updates from stylistic and narrative updates in news writing. We annotate over 9,200 pairs of sentence revisions and train high-scoring ensemble models to apply this schema. Then, taking a large dataset of silver-labeled pairs, we show that we can predict when facts will update in older article drafts with high precision. Finally, to demonstrate the usefulness of these findings, we construct a language model question asking (LLM-QA) abstention task. We wish the LLM to abstain from answering questions when information is likely to become outdated. Using our predictions, we show, LLM absention reaches near oracle levels of accuracy.