inversedMixup: Data Augmentation via Inverting Mixed Embeddings

TL;DR

InversedMixup is a novel data augmentation method that reconstructs interpretable sentences from mixed embeddings by aligning task-specific models with large language models, enhancing augmentation control and effectiveness.

Contribution

It introduces a three-stage training framework for embedding alignment, enabling interpretable mixed-sentence generation and addressing manifold intrusion in text Mixup.

Findings

Effective in few-shot and fully supervised settings

First empirical evidence of manifold intrusion in text Mixup

Improves augmentation quality and interpretability

Abstract

Mixup generates augmented samples by linearly interpolating inputs and labels with a controllable ratio. However, since it operates in the latent embedding level, the resulting samples are not human-interpretable. In contrast, LLM-based augmentation methods produce sentences via prompts at the token level, yielding readable outputs but offering limited control over the generation process. Inspired by recent advances in LLM inversion, which reconstructs natural language from embeddings and helps bridge the gap between latent embedding space and discrete token space, we propose inversedMixup, a unified framework that combines the controllability of Mixup with the interpretability of LLM-based generation. Specifically, inversedMixup adopts a three-stage training procedure to align the output embedding space of a task-specific model with the input embedding space of an LLM. Upon successful alignment, inversedMixup can reconstruct mixed embeddings with a controllable mixing ratio into human-interpretable augmented sentences, thereby improving the augmentation performance. Additionally, inversedMixup provides the first empirical evidence of the manifold intrusion phenomenon in text Mixup and introduces a simple yet effective strategy to mitigate it. Extensive experiments demonstrate the effectiveness and generalizability of our approach in both few-shot and fully supervised scenarios.