Scaling Decentralized Learning with FLock

TL;DR

FLock is a novel decentralized framework that enables secure, efficient, and robust fine-tuning of large language models in trustless environments, effectively defending against attacks and improving cross-domain generalization.

Contribution

This paper introduces FLock, integrating blockchain-based trust and economic incentives to facilitate secure, decentralized LLM fine-tuning without a central server.

Findings

Successfully fine-tuned a 70B LLM in a decentralized setting.

Reduced adversarial attack success rates by over 68%.

Achieved better cross-domain generalization compared to isolated training.

Abstract

Fine-tuning the large language models (LLMs) are prevented by the deficiency of centralized control and the massive computing and communication overhead on the decentralized schemes. While the typical standard federated learning (FL) supports data privacy, the central server requirement creates a single point of attack and vulnerability to poisoning attacks. Generalizing the result in this direction to 70B-parameter models in the heterogeneous, trustless environments has turned out to be a huge, yet unbroken bottleneck. This paper introduces FLock, a decentralized framework for secure and efficient collaborative LLM fine-tuning. Integrating a blockchain-based trust layer with economic incentives, FLock replaces the central aggregator with a secure, auditable protocol for cooperation among untrusted parties. We present the first empirical validation of fine-tuning a 70B LLM in a secure, multi-domain, decentralized setting. Our experiments show the FLock framework defends against backdoor poisoning attacks that compromise standard FL optimizers and fosters synergistic knowledge transfer. The resulting models show a >68% reduction in adversarial attack success rates. The global model also demonstrates superior cross-domain generalization, outperforming models trained in isolation on their own specialized data.