G-Mix: A Generalized Mixup Learning Framework Towards Flat Minima

TL;DR

G-Mix is a new training framework that combines Mixup and Sharpness-Aware Minimization to improve the generalization of deep neural networks, especially with limited data, by addressing manifold intrusion and optimizing sharpness sensitivity.

Contribution

The paper introduces G-Mix, a novel framework integrating Mixup and SAM, along with two algorithms, Binary G-Mix and Decomposed G-Mix, to enhance DNN generalization and address manifold intrusion.

Findings

G-Mix improves model generalization across multiple datasets.

Binary G-Mix and Decomposed G-Mix outperform existing methods.

Achieves state-of-the-art performance in experiments.

Abstract

Deep neural networks (DNNs) have demonstrated promising results in various complex tasks. However, current DNNs encounter challenges with over-parameterization, especially when there is limited training data available. To enhance the generalization capability of DNNs, the Mixup technique has gained popularity. Nevertheless, it still produces suboptimal outcomes. Inspired by the successful Sharpness-Aware Minimization (SAM) approach, which establishes a connection between the sharpness of the training loss landscape and model generalization, we propose a new learning framework called Generalized-Mixup, which combines the strengths of Mixup and SAM for training DNN models. The theoretical analysis provided demonstrates how the developed G-Mix framework enhances generalization. Additionally, to further optimize DNN performance with the G-Mix framework, we introduce two novel algorithms: Binary G-Mix and Decomposed G-Mix. These algorithms partition the training data into two subsets based on the sharpness-sensitivity of each example to address the issue of "manifold intrusion" in Mixup. Both theoretical explanations and experimental results reveal that the proposed BG-Mix and DG-Mix algorithms further enhance model generalization across multiple datasets and models, achieving state-of-the-art performance.