Deep Multimodal Transfer-Learned Regression in Data-Poor Domains

TL;DR

This paper introduces a deep multimodal transfer learning approach for regression tasks in data-scarce domains, combining pre-trained CNNs with additional feature data to improve prediction accuracy.

Contribution

The paper presents a novel deep multimodal transfer learning model that fine-tunes pre-trained CNNs with supplementary feature data for improved regression in data-poor settings.

Findings

Enhanced regression accuracy over single-mode models

Effective fine-tuning with limited image data

Successful application to microstructure image and feature data

Abstract

In many real-world applications of deep learning, estimation of a target may rely on various types of input data modes, such as audio-video, image-text, etc. This task can be further complicated by a lack of sufficient data. Here we propose a Deep Multimodal Transfer-Learned Regressor (DMTL-R) for multimodal learning of image and feature data in a deep regression architecture effective at predicting target parameters in data-poor domains. Our model is capable of fine-tuning a given set of pre-trained CNN weights on a small amount of training image data, while simultaneously conditioning on feature information from a complimentary data mode during network training, yielding more accurate single-target or multi-target regression than can be achieved using the images or the features alone. We present results using phase-field simulation microstructure images with an accompanying set of physical features, using pre-trained weights from various well-known CNN architectures, which demonstrate the efficacy of the proposed multimodal approach.