Algorithm Research of ELMo Word Embedding and Deep Learning Multimodal Transformer in Image Description

TL;DR

This paper introduces a novel zero-shot learning approach combining ELMo embeddings and deep multimodal transformers to improve medical image description and classification, addressing overfitting and semantic feature extraction issues.

Contribution

It proposes an integrated method using semantic similarity and self-attention to incorporate unknown classes and enhance feature extraction in zero-shot learning.

Findings

Achieved the highest harmonic average accuracy on three datasets.

Effectively incorporated unknown classes into the vector space.

Enhanced semantic feature extraction from medical images.

Abstract

Zero sample learning is an effective method for data deficiency. The existing embedded zero sample learning methods only use the known classes to construct the embedded space, so there is an overfitting of the known classes in the testing process. This project uses category semantic similarity measures to classify multiple tags. This enables it to incorporate unknown classes that have the same meaning as currently known classes into the vector space when it is built. At the same time, most of the existing zero sample learning algorithms directly use the depth features of medical images as input, and the feature extraction process does not consider semantic information. This project intends to take ELMo-MCT as the main task and obtain multiple visual features related to the original image through self-attention mechanism. In this paper, a large number of experiments are carried out on three zero-shot learning reference datasets, and the best harmonic average accuracy is obtained compared with the most advanced algorithms.