Confidence Optimization for Probabilistic Encoding

TL;DR

This paper introduces a confidence optimization method for probabilistic encoding in neural networks, improving distance reliability and generalization by refining variance regularization and incorporating confidence-aware mechanisms.

Contribution

It proposes a novel confidence-aware probabilistic encoding approach that replaces KL divergence with L2 regularization, enhancing model reliability and performance.

Findings

Significant performance improvements on natural language classification tasks.

Enhanced generalization on BERT and RoBERTa models.

Improved distance measurement reliability in probabilistic encoding.

Abstract

Probabilistic encoding introduces Gaussian noise into neural networks, enabling a smooth transition from deterministic to uncertain states and enhancing generalization ability. However, the randomness of Gaussian noise distorts point-based distance measurements in classification tasks. To mitigate this issue, we propose a confidence optimization probabilistic encoding (CPE) method that improves distance reliability and enhances representation learning. Specifically, we refine probabilistic encoding with two key strategies: First, we introduce a confidence-aware mechanism to adjust distance calculations, ensuring consistency and reliability in probabilistic encoding classification tasks. Second, we replace the conventional KL divergence-based variance regularization, which relies on unreliable prior assumptions, with a simpler L2 regularization term to directly constrain variance. The method we proposed is model-agnostic, and extensive experiments on natural language classification tasks demonstrate that our method significantly improves performance and generalization on both the BERT and the RoBERTa model.