A Study of "Churn" in Tweets and Real-Time Search Queries (Extended Version)

TL;DR

This paper analyzes the rapid changes in term distributions on Twitter, called 'churn', to understand their impact on real-time search ranking and system design.

Contribution

It provides the first detailed analysis of tweet and query churn, revealing insights into temporal dynamics crucial for real-time search systems.

Findings

Term distributions on Twitter change rapidly within hours.

Churn significantly affects term frequency and query statistics.

Implications for designing adaptive real-time search algorithms.

Abstract

The real-time nature of Twitter means that term distributions in tweets and in search queries change rapidly: the most frequent terms in one hour may look very different from those in the next. Informally, we call this phenomenon "churn". Our interest in analyzing churn stems from the perspective of real-time search. Nearly all ranking functions, machine-learned or otherwise, depend on term statistics such as term frequency, document frequency, as well as query frequencies. In the real-time context, how do we compute these statistics, considering that the underlying distributions change rapidly? In this paper, we present an analysis of tweet and query churn on Twitter, as a first step to answering this question. Analyses reveal interesting insights on the temporal dynamics of term distributions on Twitter and hold implications for the design of search systems.