Teach me how to Interpolate a Myriad of Embeddings

TL;DR

This paper introduces MultiMix, a novel interpolation method that combines multiple embeddings at once, improving data augmentation and class clustering in embedding space, leading to better performance on benchmarks.

Contribution

The paper proposes MultiMix, allowing interpolation of multiple tuples with individual vectors, dense spatial interpolation for sequence data, and a self-distillation approach for consistent target generation.

Findings

Significant performance improvements over state-of-the-art mixup methods.

Classes are more tightly clustered and uniformly spread in embedding space.

Enhanced data augmentation efficiency with minimal additional cost.

Abstract

Mixup refers to interpolation-based data augmentation, originally motivated as a way to go beyond empirical risk minimization (ERM). Yet, its extensions focus on the definition of interpolation and the space where it takes place, while the augmentation itself is less studied: For a mini-batch of size $m$, most methods interpolate between $m$ pairs with a single scalar interpolation factor $\lambda$. In this work, we make progress in this direction by introducing MultiMix, which interpolates an arbitrary number $n$ of tuples, each of length $m$, with one vector $\lambda$ per tuple. On sequence data, we further extend to dense interpolation and loss computation over all spatial positions. Overall, we increase the number of tuples per mini-batch by orders of magnitude at little additional cost. This is possible by interpolating at the very last layer before the classifier. Finally, to address inconsistencies due to linear target interpolation, we introduce a self-distillation approach to generate and interpolate synthetic targets. We empirically show that our contributions result in significant improvement over state-of-the-art mixup methods on four benchmarks. By analyzing the embedding space, we observe that the classes are more tightly clustered and uniformly spread over the embedding space, thereby explaining the improved behavior.