UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptation

TL;DR

UtilGen introduces a utility-centric data augmentation framework that adaptively generates task-specific synthetic data by leveraging downstream task feedback, significantly improving model performance across multiple benchmarks.

Contribution

It proposes a novel dual-level optimization strategy for generative data augmentation that focuses on maximizing task utility rather than just visual quality.

Findings

Achieves an average accuracy improvement of 3.87% over SOTA methods.

Produces more impactful, task-relevant synthetic data.

Demonstrates effectiveness across eight benchmark datasets.

Abstract

Data augmentation using generative models has emerged as a powerful paradigm for enhancing performance in computer vision tasks. However, most existing augmentation approaches primarily focus on optimizing intrinsic data attributes -- such as fidelity and diversity -- to generate visually high-quality synthetic data, while often neglecting task-specific requirements. Yet, it is essential for data generators to account for the needs of downstream tasks, as training data requirements can vary significantly across different tasks and network architectures. To address these limitations, we propose UtilGen, a novel utility-centric data augmentation framework that adaptively optimizes the data generation process to produce task-specific, high-utility training data via downstream task feedback. Specifically, we first introduce a weight allocation network to evaluate the task-specific utility of each synthetic sample. Guided by these evaluations, UtilGen iteratively refines the data generation process using a dual-level optimization strategy to maximize the synthetic data utility: (1) model-level optimization tailors the generative model to the downstream task, and (2) instance-level optimization adjusts generation policies -- such as prompt embeddings and initial noise -- at each generation round. Extensive experiments on eight benchmark datasets of varying complexity and granularity demonstrate that UtilGen consistently achieves superior performance, with an average accuracy improvement of 3.87% over previous SOTA. Further analysis of data influence and distribution reveals that UtilGen produces more impactful and task-relevant synthetic data, validating the effectiveness of the paradigm shift from visual characteristics-centric to task utility-centric data augmentation.