Relevance-driven Input Dropout: an Explanation-guided Regularization Technique

TL;DR

Relevance-driven Input Dropout (RelDrop) is a new regularization technique that selectively occludes important input regions during training, encouraging models to utilize diverse features and thereby enhancing generalization and robustness.

Contribution

The paper introduces RelDrop, a novel relevance-guided data augmentation method that improves model generalization by selectively occluding key input regions based on their relevance.

Findings

RelDrop enhances robustness to occlusion.

Models trained with RelDrop utilize more diverse features.

RelDrop improves inference time generalization.

Abstract

Overfitting is a well-known issue extending even to state-of-the-art (SOTA) Machine Learning (ML) models, resulting in reduced generalization, and a significant train-test performance gap. Mitigation measures include a combination of dropout, data augmentation, weight decay, and other regularization techniques. Among the various data augmentation strategies, occlusion is a prominent technique that typically focuses on randomly masking regions of the input during training. Most of the existing literature emphasizes randomness in selecting and modifying the input features instead of regions that strongly influence model decisions. We propose Relevance-driven Input Dropout (RelDrop), a novel data augmentation method which selectively occludes the most relevant regions of the input, nudging the model to use other important features in the prediction process, thus improving model generalization through informed regularization. We further conduct qualitative and quantitative analyses to study how Relevance-driven Input Dropout (RelDrop) affects model decision-making. Through a series of experiments on benchmark datasets, we demonstrate that our approach improves robustness towards occlusion, results in models utilizing more features within the region of interest, and boosts inference time generalization performance. Our code is available at https://github.com/Shreyas-Gururaj/LRP_Relevance_Dropout.