Learning Against Distributional Uncertainty: On the Trade-off Between Robustness and Specificity

TL;DR

This paper introduces a unified framework for trustworthy machine learning that balances robustness against distributional uncertainties with specificity to training data, addressing limitations of existing methods.

Contribution

It proposes a new model unifying Bayesian, DRO, and regularization approaches, overcoming their individual challenges and analyzing its theoretical properties.

Findings

The model demonstrates favorable asymptotic and non-asymptotic properties.

Experiments show improved performance over traditional methods.

The framework effectively balances robustness and specificity.

Abstract

Trustworthy machine learning aims at combating distributional uncertainties in training data distributions compared to population distributions. Typical treatment frameworks include the Bayesian approach, (min-max) distributionally robust optimization (DRO), and regularization. However, three issues have to be raised: 1) the prior distribution in the Bayesian method and the regularizer in the regularization method are difficult to specify; 2) the DRO method tends to be overly conservative; 3) all the three methods are biased estimators of the true optimal cost. This paper studies a new framework that unifies the three approaches and addresses the three challenges above. The asymptotic properties (e.g., consistencies and asymptotic normalities), non-asymptotic properties (e.g., generalization bounds and unbiasedness), and solution methods of the proposed model are studied. The new model reveals the trade-off between the robustness to the unseen data and the specificity to the training data. Experiments on various real-world tasks validate the superiority of the proposed learning framework.