Semantics-Aware Generative Latent Data Augmentation for Learning in Low-Resource Domains

TL;DR

This paper introduces GeLDA, a semantics-aware generative data augmentation method using diffusion models in latent space, improving performance in low-resource recognition tasks like speech emotion recognition and long-tailed image classification.

Contribution

GeLDA is a novel framework that synthesizes high-quality, semantically conditioned data in a low-dimensional latent space to enhance learning in low-resource domains.

Findings

Improves speech emotion recognition accuracy by 6.13% in zero-shot settings.

Achieves 74.7% tail-class accuracy on ImageNet-LT, setting a new state-of-the-art.

Demonstrates effectiveness of latent space augmentation in low-resource scenarios.

Abstract

Despite strong performance in data-rich regimes, deep learning often underperforms in the data-scarce settings common in practice. While foundation models (FMs) trained on massive datasets demonstrate strong generalization by extracting general-purpose features, they can still suffer from scarce labeled data during downstream fine-tuning. To address this, we propose GeLDA, a semantics-aware generative latent data augmentation framework that leverages conditional diffusion models to synthesize samples in an FM-induced latent space. Because this space is low-dimensional and concentrates task-relevant information compared to the input space, GeLDA enables efficient, high-quality data generation. GeLDA conditions generation on auxiliary feature vectors that capture semantic relationships among classes or subdomains, facilitating data augmentation in low-resource domains. We validate GeLDA in two large-scale recognition tasks: (a) in zero-shot language-specific speech emotion recognition, GeLDA improves the Whisper-large baseline's unweighted average recall by 6.13%; and (b) in long-tailed image classification, it achieves 74.7% tail-class accuracy on ImageNet-LT, setting a new state-of-the-art result.