Cross-model convolutional neural network for multiple modality data representation

TL;DR

This paper introduces a novel CNN-based method for representing multi-modality data in a unified space, improving cross-modal data analysis by leveraging relevance regularization and joint label prediction.

Contribution

It proposes a new CNN-based approach that maps different modalities into a common space with relevance regularization and label prediction, solved via ALM and ADMM.

Findings

Demonstrates improved performance on multi-modality sequence data benchmarks.

Effectively maps diverse data modalities into a shared representation space.

Enhances cross-modal data analysis with a novel CNN-based framework.

Abstract

A novel data representation method of convolutional neural net- work (CNN) is proposed in this paper to represent data of different modalities. We learn a CNN model for the data of each modality to map the data of differ- ent modalities to a common space, and regularize the new representations in the common space by a cross-model relevance matrix. We further impose that the class label of data points can also be predicted from the CNN representa- tions in the common space. The learning problem is modeled as a minimiza- tion problem, which is solved by an augmented Lagrange method (ALM) with updating rules of Alternating direction method of multipliers (ADMM). The experiments over benchmark of sequence data of multiple modalities show its advantage.