Balancing Robustness and Sensitivity using Feature Contrastive Learning

TL;DR

This paper introduces Feature Contrastive Learning (FCL), a method that balances robustness and sensitivity in large models by emphasizing features with higher contextual utility, improving generalization under noise.

Contribution

The paper proposes FCL, a novel training approach that enhances the balance between robustness and sensitivity in neural networks, addressing a key trade-off in model training.

Findings

FCL improves the robustness-sensitivity trade-off in vision and NLP models.

Models trained with FCL generalize better under noisy conditions.

FCL outperforms baseline methods in robustness and sensitivity metrics.

Abstract

It is generally believed that robust training of extremely large networks is critical to their success in real-world applications. However, when taken to the extreme, methods that promote robustness can hurt the model's sensitivity to rare or underrepresented patterns. In this paper, we discuss this trade-off between sensitivity and robustness to natural (non-adversarial) perturbations by introducing two notions: contextual feature utility and contextual feature sensitivity. We propose Feature Contrastive Learning (FCL) that encourages a model to be more sensitive to the features that have higher contextual utility. Empirical results demonstrate that models trained with FCL achieve a better balance of robustness and sensitivity, leading to improved generalization in the presence of noise on both vision and NLP datasets.