Towards Practical Few-shot Federated NLP

TL;DR

This paper introduces AUG-FedPrompt, a prompt-based federated learning system that leverages unlabeled data for data augmentation, enabling effective NLP model fine-tuning with minimal labeled data across distributed devices.

Contribution

It proposes a novel federated few-shot learning framework with a data generator and AUG-FedPrompt system utilizing unlabeled data for improved NLP performance.

Findings

AUG-FedPrompt achieves comparable results to full-set fine-tuning with limited labeled data.

The system effectively exploits unlabeled data for data augmentation in federated settings.

Performance gains are achieved at the cost of increased system complexity.

Abstract

Transformer-based pre-trained models have emerged as the predominant solution for natural language processing (NLP). Fine-tuning such pre-trained models for downstream tasks often requires a considerable amount of labeled private data. In practice, private data is often distributed across heterogeneous mobile devices and may be prohibited from being uploaded. Moreover, well-curated labeled data is often scarce, presenting an additional challenge. To address these challenges, we first introduce a data generator for federated few-shot learning tasks, which encompasses the quantity and skewness of scarce labeled data in a realistic setting. Subsequently, we propose AUG-FedPrompt, a prompt-based federated learning system that exploits abundant unlabeled data for data augmentation. Our experiments indicate that AUG-FedPrompt can perform on par with full-set fine-tuning with a limited amount of labeled data. However, such competitive performance comes at a significant system cost.