Adaptive Augmentation Policy Optimization with LLM Feedback

TL;DR

This paper introduces a novel LLM-guided augmentation policy optimization method that dynamically refines data augmentation strategies during training, improving model accuracy efficiently especially in domain-specific tasks.

Contribution

It presents a new adaptive augmentation approach using LLM feedback that reduces computational costs and enhances performance over traditional static or search-based methods.

Findings

Consistent accuracy improvements on domain-specific image datasets

Reduces computational costs by eliminating full retraining cycles

Effective in medical imaging and other specialized domains

Abstract

Data augmentation is a critical component of deep learning pipelines, enhancing model generalization by increasing dataset diversity. Traditional augmentation strategies rely on manually designed transformations, stochastic sampling, or automated search-based approaches. Although automated methods improve performance, they often require extensive computational resources and are specifically designed for certain datasets. In this work, we propose a Large Language Model (LLM)-guided augmentation optimization strategy that refines augmentation policies based on model performance feedback. We propose two approaches: (1) LLM-Guided Augmentation Policy Optimization, where augmentation policies selected by LLM are refined iteratively across training cycles, and (2) Adaptive LLM-Guided Augmentation Policy Optimization, which adjusts policies at each iteration based on performance metrics. This in-training approach eliminates the need for full model retraining before getting LLM feedback, reducing computational costs while increasing performance. Our methodology employs an LLM to dynamically select augmentation transformations based on dataset characteristics, model architecture, and prior training performance. Leveraging LLMs' contextual knowledge, especially in domain-specific tasks like medical imaging, our method selects augmentations tailored to dataset characteristics and model performance. Experiments across domain-specific image classification datasets show consistent accuracy improvements over traditional methods.