NewsRECON: News article REtrieval for image CONtextualization

TL;DR

NewsRECON is a novel method that links news images to relevant articles to determine their date and location, especially when reverse image search results are unavailable, aiding journalists and forensic experts.

Contribution

It introduces a new approach combining bi-encoders and cross-encoders to improve image-to-article linking without relying on RIS, outperforming prior methods.

Findings

Outperforms previous methods on TARA and 5Pils-OOC datasets.

Effective in the absence of reverse image search evidence.

Achieves state-of-the-art results when combined with multimodal large language models.

Abstract

Identifying when and where a news image was taken is crucial for journalists and forensic experts to produce credible stories and debunk misinformation. While many existing methods rely on reverse image search (RIS) engines, these tools often fail to return results, thereby limiting their practical applicability. In this work, we address the challenging scenario where RIS evidence is unavailable. We introduce NewsRECON, a method that links images to relevant news articles to infer their date and location from article metadata. NewsRECON leverages a corpus of over 90,000 articles and integrates: (1) a bi-encoder for retrieving event-relevant articles; (2) two cross-encoders for reranking articles by location and event consistency. Experiments on the TARA and 5Pils-OOC show that NewsRECON outperforms prior work and can be combined with a multimodal large language model to achieve new SOTA results in the absence of RIS evidence. We make our code available.