DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models

TL;DR

This paper introduces DeCAF, a novel decentralized algorithm combining Low-Rank Adaptation with matrix factorization, improving convergence and performance in training large vision-language models on edge devices.

Contribution

DeCAF integrates DLoRA with TSVD-based matrix factorization, providing theoretical guarantees and improved convergence in decentralized training of foundation models.

Findings

DeCAF matches the convergence rate of decentralized SGD.

DeCAF outperforms local training methods.

DeCAF rivals federated learning in diverse data settings.

Abstract

Low-Rank Adaptation (LoRA) has emerged as one of the most effective, computationally tractable fine-tuning approaches for training Vision-Language Models (VLMs) and Large Language Models (LLMs). LoRA accomplishes this by freezing the pre-trained model weights and injecting trainable low-rank matrices, allowing for efficient learning of these foundation models even on edge devices. However, LoRA in decentralized settings still remains under explored, particularly for the theoretical underpinnings due to the lack of smoothness guarantee and model consensus interference (defined formally below). This work improves the convergence rate of decentralized LoRA (DLoRA) to match the rate of decentralized SGD by ensuring gradient smoothness. We also introduce DeCAF, a novel algorithm integrating DLoRA with truncated singular value decomposition (TSVD)-based matrix factorization to resolve consensus interference. Theoretical analysis shows TSVD's approximation error is bounded and consensus differences between DLoRA and DeCAF vanish as rank increases, yielding DeCAF's matching convergence rate. Extensive experiments across vision/language tasks demonstrate our algorithms outperform local training and rivals federated learning under both IID and non-IID data distributions.