Mitigating Uncertainty in Document Classification

TL;DR

This paper introduces a neural network model with a novel dropout-entropy uncertainty measurement and metric learning to improve document classification accuracy, especially in scenarios involving human review of uncertain predictions.

Contribution

The paper presents a new neural network approach combining dropout-entropy and metric learning to enhance uncertainty estimation and overall accuracy in document classification.

Findings

Significant accuracy improvement on real-world datasets.

Enhanced uncertainty measurement reduces false predictions.

Effective in scenarios with human-in-the-loop classification.

Abstract

The uncertainty measurement of classifiers' predictions is especially important in applications such as medical diagnoses that need to ensure limited human resources can focus on the most uncertain predictions returned by machine learning models. However, few existing uncertainty models attempt to improve overall prediction accuracy where human resources are involved in the text classification task. In this paper, we propose a novel neural-network-based model that applies a new dropout-entropy method for uncertainty measurement. We also design a metric learning method on feature representations, which can boost the performance of dropout-based uncertainty methods with smaller prediction variance in accurate prediction trials. Extensive experiments on real-world data sets demonstrate that our method can achieve a considerable improvement in overall prediction accuracy compared to existing approaches. In particular, our model improved the accuracy from 0.78 to 0.92 when 30\% of the most uncertain predictions were handed over to human experts in "20NewsGroup" data.