FedCoT: Federated Chain-of-Thought Distillation for Large Language Models

TL;DR

FedCoT introduces a federated framework for transferring knowledge from large to small language models using Chain-of-Thought distillation, ensuring data privacy and improved performance in resource-constrained environments.

Contribution

It presents a novel federated approach with privacy-preserving strategies for Chain-of-Thought distillation from LLMs to SLMs, addressing privacy and efficiency challenges.

Findings

Enhanced SLM performance on text generation tasks

Effective privacy protection via proposed mechanisms

Feasible federated knowledge transfer in resource-limited settings

Abstract

Large Language Models (LLMs) have emerged as a transformative force in artificial intelligence, demonstrating exceptional proficiency across various tasks. However, their deployment in resource-constrained environments and concerns over user data privacy pose significant challenges. In contrast, Small Language Models (SLMs) offer computational efficiency but often lag in performance. To address these issues, we propose FedCoT, a federated framework designed for the Chain-of-Thought (CoT) distillation of knowledge from LLMs to SLMs, while ensuring the preservation of clients' data privacy. FedCoT ensures secure and efficient knowledge transfer from an LLM on a high-powered server to an SLM on a resource-constrained client, while adhering to privacy requirements. Leveraging perturbed prompts and rationales generated through the CoT approach, the framework enhances the performance of the client's SLM without compromising user data privacy within a multi-task learning framework. We propose two privacy protection strategies: the Exponential Mechanism Strategy and the Adaptive Exponential Mechanism Strategy, which balance user prompt privacy and the usability of rationales. Empirical evaluation on various text generation tasks demonstrates the effectiveness of FedCoT in training task-specific SLMs with enhanced performance while prioritizing data privacy protection. Our code has been contributed to the FATE open-source project and is now publicly accessible at \textit{https://github.com/FederatedAI/FATE-LLM/tree/main/python/fate_llm/algo/fedcot}