Batch Augmentation with Unimodal Fine-tuning for Multimodal Learning

TL;DR

This paper introduces a novel batch augmentation and unimodal fine-tuning approach for multimodal learning in medical ultrasound imaging, enhancing fetus organ detection by combining image and textual data with pre-training on medical data.

Contribution

It proposes a new training framework that integrates batch augmentation, unimodal fine-tuning, and multimodal data fusion, achieving improved performance on ultrasound datasets.

Findings

Achieved near state-of-the-art results on UPMC Food-101 dataset.

Demonstrated improved fetus organ detection accuracy.

Provided open-source scripts for the proposed method.

Abstract

This paper proposes batch augmentation with unimodal fine-tuning to detect the fetus's organs from ultrasound images and associated clinical textual information. We also prescribe pre-training initial layers with investigated medical data before the multimodal training. At first, we apply a transferred initialization with the unimodal image portion of the dataset with batch augmentation. This step adjusts the initial layer weights for medical data. Then, we apply neural networks (NNs) with fine-tuned initial layers to images in batches with batch augmentation to obtain features. We also extract information from descriptions of images. We combine this information with features obtained from images to train the head layer. We write a dataloader script to load the multimodal data and use existing unimodal image augmentation techniques with batch augmentation for the multimodal data. The dataloader brings a new random augmentation for each batch to get a good generalization. We investigate the FPU23 ultrasound and UPMC Food-101 multimodal datasets. The multimodal large language model (LLM) with the proposed training provides the best results among the investigated methods. We receive near state-of-the-art (SOTA) performance on the UPMC Food-101 dataset. We share the scripts of the proposed method with traditional counterparts at the following repository: github.com/dipuk0506/multimodal