Quantized-Tinyllava: a new multimodal foundation model enables efficient split learning

TL;DR

Quantized-TinyLLaVA introduces a communication-efficient split learning framework for multimodal models by quantizing intermediate features, significantly reducing data transmission costs while maintaining performance and enhancing privacy.

Contribution

This work presents a novel quantization-based split learning approach for multimodal models, reducing communication overhead and improving privacy in distributed training.

Findings

87.5% reduction in communication overhead with 2-bit quantization

Maintains model performance across five benchmark datasets

Enhanced resistance to feature inversion attacks

Abstract

Multimodal foundation models are increasingly trained on sensitive data across domains such as finance, biomedicine, and personal identifiers. However, this distributed setup raises serious privacy concerns due to the need for cross-partition data sharing. Split learning addresses these concerns by enabling collaborative model training without raw data exchange between partitions, yet it introduces a significant challenge: transmitting high-dimensional intermediate feature representations between partitions leads to substantial communication costs. To address this challenge, we propose Quantized-TinyLLaVA, a multimodal foundation model with an integrated communication-efficient split learning framework. Our approach adopts a compression module that quantizes intermediate feature into discrete representations before transmission, substantially reducing communication overhead. Besides, we derive a principled quantization strategy grounded in entropy coding theory to determine the optimal number of discrete representation levels. We deploy our framework in a two-partition setting, with one partition operating as the client and the other as the server, to realistically simulate distributed training. Under this setup, Quantized-TinyLLaVA achieves an approximate \textbf{87.5\%} reduction in communication overhead with 2-bit quantization, while maintaining performance of the original 16-bit model across five benchmark datasets. Furthermore, our compressed representations exhibit enhanced resilience against feature inversion attacks, validating the privacy of transmission. The code is available at https://github.com/anonymous-1742/Quantized-TinyLLaVA.