DeepDefense: Layer-Wise Gradient-Feature Alignment for Building Robust Neural Networks

TL;DR

DeepDefense introduces a layer-wise gradient-feature alignment method that enhances neural network robustness against adversarial attacks by smoothing the loss landscape and aligning gradients with internal features.

Contribution

It proposes a novel Gradient-Feature Alignment regularization across layers, improving robustness without architecture constraints.

Findings

Outperforms standard adversarial training on CIFAR-10 by up to 15.2% under APGD attacks.

Requires 20-30 times higher perturbations to fool models against DeepFool and EADEN attacks.

Achieves a flatter loss landscape and stronger decision boundaries.

Abstract

Deep neural networks are known to be vulnerable to adversarial perturbations, which are small and carefully crafted inputs that lead to incorrect predictions. In this paper, we propose DeepDefense, a novel defense framework that applies Gradient-Feature Alignment (GFA) regularization across multiple layers to suppress adversarial vulnerability. By aligning input gradients with internal feature representations, DeepDefense promotes a smoother loss landscape in tangential directions, thereby reducing the model's sensitivity to adversarial noise. We provide theoretical insights into how adversarial perturbation can be decomposed into radial and tangential components and demonstrate that alignment suppresses loss variation in tangential directions, where most attacks are effective. Empirically, our method achieves significant improvements in robustness across both gradient-based and optimization-based attacks. For example, on CIFAR-10, CNN models trained with DeepDefense outperform standard adversarial training by up to 15.2% under APGD attacks and 24.7% under FGSM attacks. Against optimization-based attacks such as DeepFool and EADEN, DeepDefense requires 20 to 30 times higher perturbation magnitudes to cause misclassification, indicating stronger decision boundaries and a flatter loss landscape. Our approach is architecture-agnostic, simple to implement, and highly effective, offering a promising direction for improving the adversarial robustness of deep learning models.