Efficient Domain Adaptation of Multimodal Embeddings using Constrastive Learning

TL;DR

This paper introduces a contrastive learning-based method to adapt multimodal embeddings for downstream tasks, achieving high performance with minimal computational resources, especially suitable for resource-constrained environments like healthcare.

Contribution

The authors propose a novel approach that adapts foundational multimodal embeddings using contrastive learning without fine-tuning large models, making advanced ML accessible in resource-limited settings.

Findings

Significant performance improvements on downstream tasks.

Minimal computational overhead compared to fine-tuning.

Effective adaptation of frozen embeddings for practical applications.

Abstract

Recent advancements in machine learning (ML), natural language processing (NLP), and foundational models have shown promise for real-life applications in critical, albeit compute-constrainted fields like healthcare. In such areas, combining foundational models with supervised ML offers potential for automating tasks like diagnosis and treatment planning, but the limited availability of onsite computational resources pose significant challenges before applying these technologies effectively: Current approaches either yield subpar results when using pretrained models without task-specific adaptation, or require substantial computational resources for fine-tuning, which is often a barrier to entry in such environments. This renders them inaccessible in applications where performance and quality standards are high, but computational resources are scarce. To bridge the gap between best-in-class performance and accessibility, we propose a novel method for adapting foundational, multimodal embeddings to downstream tasks, without the need of expensive fine-tuning processes. Our method leverages frozen embeddings from Large Language Models (LLMs) and Vision Models, and uses contrastive learning to train a small, task-specific nonlinear projection that can be used in the downstream task, without having to fine-tune the original foundational models. We show that this efficient procedure leads to significant performance improvements across various downstream tasks, and perhaps more importantly with minimal computational overhead, offering a practical solution for the use of advanced, foundational ML models in resource-constrained settings.