UPLME: Uncertainty-Aware Probabilistic Language Modelling for Robust Empathy Regression

TL;DR

UPLME introduces an uncertainty-aware probabilistic language model for empathy regression that effectively handles noisy labels, achieving state-of-the-art results and improved uncertainty quantification on benchmark datasets.

Contribution

The paper proposes UPLME, a novel probabilistic language modeling framework that captures label noise and heteroscedastic uncertainty in empathy regression tasks using Bayesian methods.

Findings

State-of-the-art Pearson correlation coefficients on benchmarks.

Effective distinction between noisy and clean samples.

Improved calibration error over existing UQ methods.

Abstract

Noisy self-reported empathy scores challenge supervised learning for empathy regression. While many algorithms have been proposed for learning with noisy labels in textual classification problems, the regression counterpart is relatively under-explored. We propose UPLME, an uncertainty-aware probabilistic language modelling framework to capture label noise in empathy regression tasks. One of the novelties in UPLME is a probabilistic language model that predicts both empathy scores and heteroscedastic uncertainty, and is trained using Bayesian concepts with variational model ensembling. We further introduce two novel loss components: one penalises degenerate Uncertainty Quantification (UQ), and another enforces similarity between the input pairs on which empathy is being predicted. UPLME achieves state-of-the-art performance (Pearson Correlation Coefficient: $0.558\rightarrow0.580$ and $0.629\rightarrow0.634$) in terms of the performance reported in the literature on two public benchmarks with label noise. Through synthetic label noise injection, we demonstrate that UPLME is effective in distinguishing between noisy and clean samples based on the predicted uncertainty. UPLME further outperform (Calibration error: $0.571\rightarrow0.376$) a recent variational model ensembling-based UQ method designed for regression problems. Code is publicly available at https://github.com/hasan-rakibul/UPLME.