FDRMFL:Multi-modal Federated Feature Extraction Model Based on Information Maximization and Contrastive Learning

TL;DR

This paper introduces a federated multi-modal feature extraction method that leverages information maximization and contrastive learning to improve regression accuracy in non-IID, limited data scenarios while mitigating catastrophic forgetting.

Contribution

It proposes a novel task-driven federated feature extraction framework integrating multi-modal information, contrastive learning, and multi-constraint optimization for improved regression performance.

Findings

Outperforms classical feature extraction techniques in regression tasks.

Effectively mitigates catastrophic forgetting in non-IID data.

Enhances multi-modal feature fusion and alignment.

Abstract

This study focuses on the feature extraction problem in multi-modal data regression. To address three core challenges in real-world scenarios: limited and non-IID data, effective extraction and fusion of multi-modal information, and susceptibility to catastrophic forgetting in model learning, a task-driven supervised multi-modal federated feature extraction method is proposed. The method integrates multi-modal information extraction and contrastive learning mechanisms, and can adapt to different neural network structures as the latent mapping functions for data of each modality. It supports each client to independently learn low-dimensional representations of multi-modal data, and can flexibly control the degree of retention of effective information about the response variable in the predictive variables within the low-dimensional features through parameter tuning. The multi-constraint learning framework constructed by the method guarantees regression accuracy using Mean Squared Error loss. Through the synergistic effect of mutual information preservation constraint, symmetric Kullback-Leibler divergence constraint, and inter-model contrastive constraint, it achieves the retention of task-related information, the extraction, fusion, and alignment of multi-modal features, and the mitigation of representation drift and catastrophic forgetting in non-IID scenarios, respectively. This ensures that the feature extraction process always centers on improving the performance of downstream regression tasks. Experimental results from simulations and real-world data analysis demonstrate that the proposed method achieves more significant performance improvement on downstream regression tasks compared with classical feature extraction techniques.