ML-ECS: A Collaborative Multimodal Learning Framework for Edge-Cloud Synergies

TL;DR

ML-ECS is a novel framework enabling collaborative multimodal learning between edge devices and cloud servers, effectively handling heterogeneity and improving performance while maintaining high communication efficiency.

Contribution

It introduces a comprehensive framework with four components that address modality and model heterogeneity in edge-cloud multimodal learning environments.

Findings

Outperforms state-of-the-art baselines by 5.44% to 12.08% in Rouge-LSum.

Achieves high communication efficiency with only 0.65% of total parameters communicated.

Enhances both client and server performance across various multimodal tasks.

Abstract

Edge-cloud synergies provide a promising paradigm for privacy-preserving deployment of foundation models, where lightweight on-device models adapt to domain-specific data and cloud-hosted models coordinate knowledge sharing. However, in real-world edge environments, collaborative multimodal learning is challenged by modality heterogeneity (different modality combinations across domains) and model-structure heterogeneity (different modality-specific encoders/fusion modules. To address these issues, we propose ML-ECS, a collaborative multimodal learning framework that enables joint training between a server-based model and heterogeneous edge models. This framework consists of four components: (1) cross-modal contrastive learning (CCL) to align modality representations in a shared latent space, (2) adaptive multimodal tuning (AMT) to preserve domain-specific knowledge from local datasets, (3) modality-aware model aggregation (MMA) to robustly aggregate while mitigating noise caused by missing modalities, and (4) SLM-enhanced CCL (SE-CCL) to facilitate bidirectional knowledge transfer between cloud and edge. Experimental results on various multimodal tasks show that \pname consistently outperform state-of-the-art baselines under varying modality availability, achieving improvements of 5.44% to 12.08% in Rouge-LSum and improving both client- and server-side performance. In addition, by communicating only low-rank LoRA parameters and fused representations, ML-ECS achieves high communication efficiency, requiring only 0.65% of the total parameter volume.