Kernel-convoluted Deep Neural Networks with Data Augmentation

TL;DR

This paper introduces kernel-convoluted models combined with data augmentation via Mixup to enhance smoothness, generalization, and adversarial robustness in deep neural networks, supported by theoretical analysis and empirical results.

Contribution

It proposes a new class of kernel-convoluted models with explicit smoothness constraints and extends them with Mixup, providing risk analysis and demonstrating improved performance.

Findings

KCM with Mixup outperforms standard Mixup in generalization.

Theoretical risk bounds show no slower convergence than original models.

Empirical results on CIFAR datasets confirm improved robustness and accuracy.

Abstract

The Mixup method (Zhang et al. 2018), which uses linearly interpolated data, has emerged as an effective data augmentation tool to improve generalization performance and the robustness to adversarial examples. The motivation is to curtail undesirable oscillations by its implicit model constraint to behave linearly at in-between observed data points and promote smoothness. In this work, we formally investigate this premise, propose a way to explicitly impose smoothness constraints, and extend it to incorporate with implicit model constraints. First, we derive a new function class composed of kernel-convoluted models (KCM) where the smoothness constraint is directly imposed by locally averaging the original functions with a kernel function. Second, we propose to incorporate the Mixup method into KCM to expand the domains of smoothness. In both cases of KCM and the KCM adapted with the Mixup, we provide risk analysis, respectively, under some conditions for kernels. We show that the upper bound of the excess risk is not slower than that of the original function class. The upper bound of the KCM with the Mixup remains dominated by that of the KCM if the perturbation of the Mixup vanishes faster than \(O(n^{-1/2})\) where \(n\) is a sample size. Using CIFAR-10 and CIFAR-100 datasets, our experiments demonstrate that the KCM with the Mixup outperforms the Mixup method in terms of generalization and robustness to adversarial examples.