Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Tasks

TL;DR

This paper introduces a $k$-Nearest Neighbor Uncertainty Estimation method for NLP tasks that improves calibration and uncertainty estimation by leveraging neighbor distances and label ratios, with efficient approximate search techniques.

Contribution

The paper proposes a novel $k$NN-UE method that enhances uncertainty estimation in NLP by combining neighbor distances and label ratios, outperforming existing methods.

Findings

Outperforms baselines and recent density-based methods in calibration metrics.

Approximate nearest neighbor search reduces inference time without significant performance loss.

Effective across multiple NLP tasks like sentiment analysis, NLI, and NER.

Abstract

Trustworthiness in model predictions is crucial for safety-critical applications in the real world. However, deep neural networks often suffer from the issues of uncertainty estimation, such as miscalibration. In this study, we propose $k$-Nearest Neighbor Uncertainty Estimation ($k$NN-UE), which is a new uncertainty estimation method that uses not only the distances from the neighbors, but also the ratio of labels in the neighbors. Experiments on sentiment analysis, natural language inference, and named entity recognition show that our proposed method outperforms the baselines and recent density-based methods in several calibration and uncertainty metrics. Moreover, our analyses indicate that approximate nearest neighbor search techniques reduce the inference overhead without significantly degrading the uncertainty estimation performance when they are appropriately combined.