Synthetic Feature Augmentation Improves Generalization Performance of Language Models

TL;DR

This paper introduces a synthetic feature augmentation method that generates artificial samples in embedding space to address data imbalance, improving the generalization and robustness of large language models on text classification tasks.

Contribution

The paper proposes a novel embedding space augmentation technique that enhances model performance on imbalanced datasets, a significant advancement over existing data balancing methods.

Findings

Improved classification accuracy on imbalanced benchmarks

Enhanced model robustness and generalization

Effective upsampling of minority classes

Abstract

Training and fine-tuning deep learning models, especially large language models (LLMs), on limited and imbalanced datasets poses substantial challenges. These issues often result in poor generalization, where models overfit to dominant classes and underperform on minority classes, leading to biased predictions and reduced robustness in real-world applications. To overcome these challenges, we propose augmenting features in the embedding space by generating synthetic samples using a range of techniques. By upsampling underrepresented classes, this method improves model performance and alleviates data imbalance. We validate the effectiveness of this approach across multiple open-source text classification benchmarks, demonstrating its potential to enhance model robustness and generalization in imbalanced data scenarios.