Multi-Modal Coreference Resolution with the Correlation between Space Structures

TL;DR

This paper introduces a semi-supervised multi-modal coreference resolution method that leverages intrinsic space structure correlations between modalities, improving performance with limited training data.

Contribution

It proposes a novel approach that uses space structure correlation for semi-supervised learning in multi-modal coreference resolution, reducing dependence on large training datasets.

Findings

Outperforms existing methods with limited training data

Uses shared reference points to correlate space structures

Effective in multi-modal datasets

Abstract

Multi-modal data is becoming more common in big data background. Finding the semantically similar objects from different modality is one of the heart problems of multi-modal learning. Most of the current methods try to learn the inter-modal correlation with extrinsic supervised information, while intrinsic structural information of each modality is neglected. The performance of these methods heavily depends on the richness of training samples. However, obtaining the multi-modal training samples is still a labor and cost intensive work. In this paper, we bring a extrinsic correlation between the space structures of each modalities in coreference resolution. With this correlation, a semi-supervised learning model for multi-modal coreference resolution is proposed. We firstly extract high-level features of images and text, then compute the distances of each object from some reference points to build the space structure of each modality. With a shared reference point set, the space structures of each modality are correlated. We employ the correlation to build a commonly shared space that the semantic distance between multi-modal objects can be computed directly. The experiments on two multi-modal datasets show that our model performs better than the existing methods with insufficient training data.