Bridging the Gap between Text, Audio, Image, and Any Sequence: A Novel Approach using Gloss-based Annotation

TL;DR

This paper introduces BGTAI, a novel multimodal framework that uses gloss-based annotations to improve alignment and understanding across text, audio, and images, enhancing multimodal representation quality.

Contribution

It proposes the first Langue2Gloss model and integrates it into UniBriVL, along with new modules and loss functions to improve multimodal alignment and training stability.

Findings

Outperforms previous multimodal models in experiments

Enhances compatibility among text, audio, and visual modalities

Demonstrates improved multimodal representations

Abstract

This paper presents an innovative approach called BGTAI to simplify multimodal understanding by utilizing gloss-based annotation as an intermediate step in aligning Text and Audio with Images. While the dynamic temporal factors in textual and audio inputs contain various predicate adjectives that influence the meaning of the entire sentence, images, on the other hand, present static scenes. By representing text and audio as gloss notations that omit complex semantic nuances, a better alignment with images can potentially be achieved. This study explores the feasibility of this idea, specifically, we first propose the first Langue2Gloss model and then integrate it into the multimodal model UniBriVL for joint training. To strengthen the adaptability of gloss with text/audio and overcome the efficiency and instability issues in multimodal training, we propose a DS-Net (Data-Pair Selection Network), an Result Filter module, and a novel SP-Loss function. Our approach outperforms previous multimodal models in the main experiments, demonstrating its efficacy in enhancing multimodal representations and improving compatibility among text, audio, visual, and any sequence modalities.