Survey: Transformer-based Models in Data Modality Conversion

TL;DR

This survey reviews transformer-based models for converting data between text, vision, and speech modalities, highlighting their architectures, methods, and applications to demonstrate their versatility in AI content generation.

Contribution

It provides a comprehensive, systematic review of transformer models for modality conversion, filling a gap in existing literature.

Findings

Transformers are effective across multiple data modalities.

Various architectures and methods have been developed for modality conversion.

Transformers significantly enhance AI content understanding and generation.

Abstract

Transformers have made significant strides across various artificial intelligence domains, including natural language processing, computer vision, and audio processing. This success has naturally garnered considerable interest from both academic and industry researchers. Consequently, numerous Transformer variants (often referred to as X-formers) have been developed for these fields. However, a thorough and systematic review of these modality-specific conversions remains lacking. Modality Conversion involves the transformation of data from one form of representation to another, mimicking the way humans integrate and interpret sensory information. This paper provides a comprehensive review of transformer-based models applied to the primary modalities of text, vision, and speech, discussing their architectures, conversion methodologies, and applications. By synthesizing the literature on modality conversion, this survey aims to underline the versatility and scalability of transformers in advancing AI-driven content generation and understanding.