The Nature of Novelty Detection

TL;DR

This paper investigates the fundamental nature of sentence-level novelty detection, revealing it as a combination of partial and complete overlap relations, and proposes new methods and evaluation metrics based on this understanding.

Contribution

It formalizes novelty detection as a combination of partial and complete overlaps, introduces the selected pool method, and proposes new evaluation measures aligned with this understanding.

Findings

Selected pool method outperforms existing methods

Novel evaluation metrics improve assessment accuracy

Task understanding influences evaluation approaches

Abstract

Sentence level novelty detection aims at reducing redundant sentences from a sentence list. In the task, sentences appearing later in the list with no new meanings are eliminated. Aiming at a better accuracy for detecting redundancy, this paper reveals the nature of the novelty detection task currently overlooked by the Novelty community $-$ Novelty as a combination of the partial overlap (PO, two sentences sharing common facts) and complete overlap (CO, the first sentence covers all the facts of the second sentence) relations. By formalizing novelty detection as a combination of the two relations between sentences, new viewpoints toward techniques dealing with Novelty are proposed. Among the methods discussed, the similarity, overlap, pool and language modeling approaches are commonly used. Furthermore, a novel approach, selected pool method is provided, which is immediate following the nature of the task. Experimental results obtained on all the three currently available novelty datasets showed that selected pool is significantly better or no worse than the current methods. Knowledge about the nature of the task also affects the evaluation methodologies. We propose new evaluation measures for Novelty according to the nature of the task, as well as possible directions for future study.