Multi-Layer Ranking with Large Language Models for News Source Recommendation

TL;DR

This paper introduces a new expert recommendation task for news sources, utilizing a large language model-based multi-layer ranking framework and a novel dataset, significantly enhancing recommendation accuracy.

Contribution

The paper presents a novel dataset and a multi-layer ranking framework employing large language models for expert recommendation in news sources.

Findings

In-context learning LLM ranker improves recommendation accuracy.

Multi-layer ranking filter enhances predictive and behavioral quality.

The approach outperforms baseline methods in expert retrieval tasks.

Abstract

To seek reliable information sources for news events, we introduce a novel task of expert recommendation, which aims to identify trustworthy sources based on their previously quoted statements. To achieve this, we built a novel dataset, called NewsQuote, consisting of 23,571 quote-speaker pairs sourced from a collection of news articles. We formulate the recommendation task as the retrieval of experts based on their likelihood of being associated with a given query. We also propose a multi-layer ranking framework employing Large Language Models to improve the recommendation performance. Our results show that employing an in-context learning based LLM ranker and a multi-layer ranking-based filter significantly improve both the predictive quality and behavioural quality of the recommender system.