Content-Based Quality Estimation for Automatic Subject Indexing of Short Texts under Precision and Recall Constraints

TL;DR

This paper introduces a deep learning-based method to estimate the quality of semantic annotations in short texts, focusing on improving document-level recall while maintaining precision under constraints.

Contribution

It presents a novel approach that detects documents meeting quality criteria using content-based indicators, enhancing document-level quality estimation in short texts.

Findings

Effective in short text collections from law and economics

Improves document-level recall without sacrificing precision

Enables filtering for high-quality data in retrieval systems

Abstract

Semantic annotations have to satisfy quality constraints to be useful for digital libraries, which is particularly challenging on large and diverse datasets. Confidence scores of multi-label classification methods typically refer only to the relevance of particular subjects, disregarding indicators of insufficient content representation at the document-level. Therefore, we propose a novel approach that detects documents rather than concepts where quality criteria are met. Our approach uses a deep, multi-layered regression architecture, which comprises a variety of content-based indicators. We evaluated multiple configurations using text collections from law and economics, where the available content is restricted to very short texts. Notably, we demonstrate that the proposed quality estimation technique can determine subsets of the previously unseen data where considerable gains in document-level recall can be achieved, while upholding precision at the same time. Hence, the approach effectively performs a filtering that ensures high data quality standards in operative information retrieval systems.