Everything You Always Wanted to Know About TREC RTS* (*But Were Afraid to Ask)

TL;DR

This paper thoroughly analyzes the TREC RTS framework from 2016-2017, identifying limitations in metrics and providing recommendations for fair reuse of the Twitter stream evaluation collection.

Contribution

It offers a detailed critique of the TREC RTS framework components and suggests improvements for more reliable real-time Twitter stream evaluation.

Findings

Metrics have limitations in current evaluation framework

Recommendations for fair reuse of the Twitter collection

Identified weaknesses in Scenario A of the track

Abstract

The TREC Real-Time Summarization (RTS) track provides a framework for evaluating systems monitoring the Twitter stream and pushing tweets to users according to given profiles. It includes metrics, files, settings and hypothesis provided by the organizers. In this work, we perform a thorough analysis of each component of the framework used in 2016 and 2017 and found some limitations for the Scenario A of this track. Our main findings point out the weakness of the metrics and give clear recommendations to fairly reuse the collection.