Learn to Accumulate Evidence from All Training Samples: Theory and Practice

TL;DR

This paper investigates the limitations of evidential deep learning models, identifies the cause of their inferior performance, and proposes a new regularizer to improve their effectiveness on large-scale datasets.

Contribution

The paper provides a theoretical analysis of evidential models, uncovers a fundamental limitation related to zero evidence regions, and introduces a novel regularizer to address this issue.

Findings

Theoretical analysis explains the cause of poor performance in evidential models.

Proposed regularizer alleviates the zero evidence region problem.

Experimental results show improved performance on real-world datasets.

Abstract

Evidential deep learning, built upon belief theory and subjective logic, offers a principled and computationally efficient way to turn a deterministic neural network uncertainty-aware. The resultant evidential models can quantify fine-grained uncertainty using the learned evidence. To ensure theoretically sound evidential models, the evidence needs to be non-negative, which requires special activation functions for model training and inference. This constraint often leads to inferior predictive performance compared to standard softmax models, making it challenging to extend them to many large-scale datasets. To unveil the real cause of this undesired behavior, we theoretically investigate evidential models and identify a fundamental limitation that explains the inferior performance: existing evidential activation functions create zero evidence regions, which prevent the model to learn from training samples falling into such regions. A deeper analysis of evidential activation functions based on our theoretical underpinning inspires the design of a novel regularizer that effectively alleviates this fundamental limitation. Extensive experiments over many challenging real-world datasets and settings confirm our theoretical findings and demonstrate the effectiveness of our proposed approach.