FuseGen: PLM Fusion for Data-generation based Zero-shot Learning

TL;DR

FuseGen introduces a multi-PLM, feedback-driven data generation framework that significantly improves zero-shot learning by enhancing synthetic dataset quality and reducing distribution bias.

Contribution

It presents a novel subset selection and iterative feedback mechanism using multiple PLMs and trained STMs to improve synthetic data quality for zero-shot learning.

Findings

Outperforms existing methods across diverse tasks.

Effectively reduces distribution bias in synthetic datasets.

Enhances STM performance in a PLM-agnostic manner.

Abstract

Data generation-based zero-shot learning, although effective in training Small Task-specific Models (STMs) via synthetic datasets generated by Pre-trained Language Models (PLMs), is often limited by the low quality of such synthetic datasets. Previous solutions have primarily focused on single PLM settings, where synthetic datasets are typically restricted to specific sub-spaces and often deviate from real-world distributions, leading to severe distribution bias. To mitigate such bias, we propose FuseGen, a novel data generation-based zero-shot learning framework that introduces a new criteria for subset selection from synthetic datasets via utilizing multiple PLMs and trained STMs. The chosen subset provides in-context feedback to each PLM, enhancing dataset quality through iterative data generation. Trained STMs are then used for sample re-weighting as well, further improving data quality. Extensive experiments across diverse tasks demonstrate that FuseGen substantially outperforms existing methods, highly effective in boosting STM performance in a PLM-agnostic way. Code is provided in https://github.com/LindaLydia/FuseGen.