GATS: Gather-Attend-Scatter

TL;DR

GATS is a flexible module that integrates pretrained models into larger multimodal systems, allowing for efficient processing across modalities without fine-tuning the original models.

Contribution

The paper introduces GATS, a novel module enabling seamless combination of pretrained models into multimodal networks, maintaining their knowledge while processing multiple modalities.

Findings

GATS effectively combines pretrained models in multimodal tasks.

GATS maintains model knowledge without fine-tuning.

Experiments demonstrate versatility across domains.

Abstract

As the AI community increasingly adopts large-scale models, it is crucial to develop general and flexible tools to integrate them. We introduce Gather-Attend-Scatter (GATS), a novel module that enables seamless combination of pretrained foundation models, both trainable and frozen, into larger multimodal networks. GATS empowers AI systems to process and generate information across multiple modalities at different rates. In contrast to traditional fine-tuning, GATS allows for the original component models to remain frozen, avoiding the risk of them losing important knowledge acquired during the pretraining phase. We demonstrate the utility and versatility of GATS with a few experiments across games, robotics, and multimodal input-output systems.