Sequential Compression Layers for Efficient Federated Learning in Foundational Models

TL;DR

This paper introduces a new parameter-efficient fine-tuning method for federated learning of large models, replacing LoRA with a small MLP layer, leading to better performance in language and vision tasks.

Contribution

A novel MLP-based fine-tuning approach that outperforms LoRA in federated learning settings for large language and vision models.

Findings

Outperforms LoRA-based methods in federated fine-tuning

Effective for both language models and vision encoders

Addresses LoRA's bottlenecks in federated settings

Abstract

Federated Learning (FL) has gained popularity for fine-tuning large language models (LLMs) across multiple nodes, each with its own private data. While LoRA has been widely adopted for parameter efficient federated fine-tuning, recent theoretical and empirical studies highlight its suboptimal performance in the federated learning context. In response, we propose a novel, simple, and more effective parameter-efficient fine-tuning method that does not rely on LoRA. Our approach introduces a small multi-layer perceptron (MLP) layer between two existing MLP layers the up proj (the FFN projection layer following the self-attention module) and down proj within the feed forward network of the transformer block. This solution addresses the bottlenecks associated with LoRA in federated fine tuning and outperforms recent LoRA-based approaches, demonstrating superior performance for both language models and vision encoders.