Self-reinforcing Unsupervised Matching

TL;DR

This paper introduces SUM, a self-reinforcing unsupervised matching method that annotates images in new modalities without supervision, leveraging cross-modality matching to improve generalization and facilitate continual learning.

Contribution

The paper proposes a novel unsupervised approach for cross-modality image annotation that requires only one template and no supervision in the emerging modality.

Findings

Enables annotation in new modalities without supervision

Requires only one template in seen modality

Facilitates continual learning in deep models

Abstract

Remarkable gains in deep learning usually rely on tremendous supervised data. Ensuring the modality diversity for one object in training set is critical for the generalization of cutting-edge deep models, but it burdens human with heavy manual labor on data collection and annotation. In addition, some rare or unexpected modalities are new for the current model, causing reduced performance under such emerging modalities. Inspired by the achievements in speech recognition, psychology and behavioristics, we present a practical solution, self-reinforcing unsupervised matching (SUM), to annotate the images with 2D structure-preserving property in an emerging modality by cross-modality matching. This approach requires no any supervision in emerging modality and only one template in seen modality, providing a possible route towards continual learning.