How to detect novelty in textual data streams? A comparative study of existing methods

TL;DR

This paper introduces a simulation framework and benchmark to evaluate existing novelty detection methods in textual data streams, addressing the lack of annotated datasets and testing various scenarios.

Contribution

It provides a novel simulation framework and comprehensive benchmark for evaluating novelty detection methods in textual streams, including real-world dataset experiments.

Findings

Benchmark results show varying performance across methods

Sensitivity analysis reveals parameter impacts on detection accuracy

Real-world experiments validate the simulation framework's effectiveness

Abstract

Since datasets with annotation for novelty at the document and/or word level are not easily available, we present a simulation framework that allows us to create different textual datasets in which we control the way novelty occurs. We also present a benchmark of existing methods for novelty detection in textual data streams. We define a few tasks to solve and compare several state-of-the-art methods. The simulation framework allows us to evaluate their performances according to a set of limited scenarios and test their sensitivity to some parameters. Finally, we experiment with the same methods on different kinds of novelty in the New York Times Annotated Dataset.