The Effects of Latency Penalties in Evaluating Push Notification Systems

TL;DR

This paper investigates how various latency penalties impact the evaluation of push notification systems, aiming to improve metric design for real-time summarization tasks.

Contribution

It provides an analysis of latency penalty effects in push notification evaluation, informing future metric development for real-time summarization.

Findings

Latency penalties significantly influence system rankings

Evaluation metrics need to account for latency effects

Guidelines for designing better real-time evaluation metrics

Abstract

We examine the effects of different latency penalties in the evaluation of push notification systems, as operationalized in the TREC 2015 Microblog track evaluation. The purpose of this study is to inform the design of metrics for the TREC 2016 Real-Time Summarization track, which is largely modeled after the TREC 2015 evaluation design.