Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond

TL;DR

This tutorial reviews recent developments in multimodal large models that integrate diverse data types like vision, language, audio, and sensors, highlighting datasets, models, and tuning strategies for practical applications.

Contribution

It provides a comprehensive overview of multimodal pretrained models, datasets, and instruction tuning techniques, including hands-on labs for real-world multimodal AI applications.

Findings

Advancements in multimodal datasets and pretrained models.

Effective instruction tuning strategies for multimodal tasks.

Practical demonstrations of multimodal applications like visual storytelling.

Abstract

This tutorial explores recent advancements in multimodal pretrained and large models, capable of integrating and processing diverse data forms such as text, images, audio, and video. Participants will gain an understanding of the foundational concepts of multimodality, the evolution of multimodal research, and the key technical challenges addressed by these models. We will cover the latest multimodal datasets and pretrained models, including those beyond vision and language. Additionally, the tutorial will delve into the intricacies of multimodal large models and instruction tuning strategies to optimise performance for specific tasks. Hands-on laboratories will offer practical experience with state-of-the-art multimodal models, demonstrating real-world applications like visual storytelling and visual question answering. This tutorial aims to equip researchers, practitioners, and newcomers with the knowledge and skills to leverage multimodal AI. ACM Multimedia 2024 is the ideal venue for this tutorial, aligning perfectly with our goal of understanding multimodal pretrained and large language models, and their tuning mechanisms.