ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization

TL;DR

ZeroPrompt demonstrates that scaling to 1,000 tasks significantly enhances zero-shot generalization and training efficiency, with minimal dependence on model size, through a novel multitask pretraining and prompt search approach.

Contribution

The paper introduces ZeroPrompt, a multitask pretraining method on 1,000 tasks, showing task scaling as an efficient alternative to model scaling and incorporating a genetic algorithm for prompt optimization.

Findings

Task scaling improves training efficiency by 30 times in FLOPs.

ZeroPrompt enhances zero-shot performance across diverse datasets.

Model size has little impact when scaling to many tasks.

Abstract

We propose a multitask pretraining approach ZeroPrompt for zero-shot generalization, focusing on task scaling and zero-shot prompting. While previous models are trained on only a few dozen tasks, we scale to 1,000 tasks for the first time using real-world data. This leads to a crucial discovery that task scaling can be an efficient alternative to model scaling; i.e., the model size has little impact on performance with an extremely large number of tasks. Our results show that task scaling can substantially improve training efficiency by 30 times in FLOPs. Moreover, we present a prompting method that incorporates a genetic algorithm to automatically search for the best prompt for unseen tasks, along with a few other improvements. Empirically, ZeroPrompt substantially improves both the efficiency and the performance of zero-shot learning across a variety of academic and production datasets.