Supervised cross-modal factor analysis for multiple modal data classification

TL;DR

This paper introduces a supervised cross-modal factor analysis method that jointly learns shared data representations and classifiers for image and text data, improving document classification accuracy.

Contribution

It extends existing CFA by incorporating supervision, enabling joint learning of data projections and classifiers for multimodal document classification.

Findings

The proposed method outperforms other CFA approaches in experiments.

Joint learning of projections and classifiers improves classification accuracy.

The algorithm effectively minimizes projection distance and classification error.

Abstract

In this paper we study the problem of learning from multiple modal data for purpose of document classification. In this problem, each document is composed two different modals of data, i.e., an image and a text. Cross-modal factor analysis (CFA) has been proposed to project the two different modals of data to a shared data space, so that the classification of a image or a text can be performed directly in this space. A disadvantage of CFA is that it has ignored the supervision information. In this paper, we improve CFA by incorporating the supervision information to represent and classify both image and text modals of documents. We project both image and text data to a shared data space by factor analysis, and then train a class label predictor in the shared space to use the class label information. The factor analysis parameter and the predictor parameter are learned jointly by solving one single objective function. With this objective function, we minimize the distance between the projections of image and text of the same document, and the classification error of the projection measured by hinge loss function. The objective function is optimized by an alternate optimization strategy in an iterative algorithm. Experiments in two different multiple modal document data sets show the advantage of the proposed algorithm over other CFA methods.