To Predict or Not to Predict? Towards reliable uncertainty estimation in the presence of noise

TL;DR

This paper evaluates various uncertainty estimation methods in multilingual text classification under noisy conditions, highlighting Monte Carlo dropout's robustness and demonstrating improved performance when abstaining from uncertain predictions.

Contribution

It provides a comprehensive comparison of UE techniques across languages and noise levels, emphasizing Monte Carlo dropout's effectiveness for reliable NLP predictions.

Findings

Monte Carlo dropout offers consistent robustness across languages.

Softmax-based methods decline in reliability under domain shift.

Abstaining from uncertain predictions improves macro F1 score from 0.81 to 0.85.

Abstract

This study examines the role of uncertainty estimation (UE) methods in multilingual text classification under noisy and non-topical conditions. Using a complex-vs-simple sentence classification task across several languages, we evaluate a range of UE techniques against a range of metrics to assess their contribution to making more robust predictions. Results indicate that while methods relying on softmax outputs remain competitive in high-resource in-domain settings, their reliability declines in low-resource or domain-shift scenarios. In contrast, Monte Carlo dropout approaches demonstrate consistently strong performance across all languages, offering more robust calibration, stable decision thresholds, and greater discriminative power even under adverse conditions. We further demonstrate the positive impact of UE on non-topical classification: abstaining from predicting the 10\% most uncertain instances increases the macro F1 score from 0.81 to 0.85 in the Readme task. By integrating UE with trustworthiness metrics, this study provides actionable insights for developing more reliable NLP systems in real-world multilingual environments. See https://github.com/Nouran-Khallaf/To-Predict-or-Not-to-Predict