Data Quality Control in Federated Instruction-tuning of Large Language Models

TL;DR

This paper introduces FedDQC, a federated instruction tuning framework with dynamic data quality control that improves large language model performance by filtering and progressively training on high-quality data.

Contribution

The paper presents a novel framework with instruction-response alignment and hierarchical training for dynamic data quality management in federated learning of LLMs.

Findings

Significant performance improvements on mixed-quality data.

Effective client-side data quality evaluation metric.

Progressive training from high- to low-IRA data enhances learning.

Abstract

Federated Learning (FL) enables privacy-preserving collaborative instruction tuning of large language models (LLMs) by leveraging massively distributed data. However, the decentralized nature of FL exacerbates data quality challenges, as local clients lack global visibility to filter noisy or low-quality samples before training. To resolve this issue, we propose FedDQC, a novel federated instruction tuning framework with dynamic data quality control. Our approach introduces two key innovations. First, we propose instruction-response alignment (IRA), an efficient client-side metric for quality evaluation requiring only low-cost inference. We validate that higher-IRA data corresponds to more relevant and easier-to-learn question-answer pairs. Second, mirroring the human easy-to-hard knowledge acquisition process, we design a quality-aware hierarchical FL training framework, where the LLM is progressively fine-tuned from high- to low-IRA data in a collaborative manner. The framework also supports adaptive data quality assessment at each hierarchy, enabling dynamic adjustments throughout the training process. Extensive experiments on synthetic and real-world datasets show that our method significantly improves LLM performance on mixed-quality data in FL.