Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception

TL;DR

This paper presents a method to align inertial measurement unit (IMU) data with vision-language models, enabling improved robot perception and multi-modal scene understanding without retraining the entire model.

Contribution

The authors propose a novel approach to align additional modalities like IMU data with vision embeddings, enhancing VLM capabilities in robot perception without retraining.

Findings

Multi-modal alignment improves scene understanding accuracy.

Using IMU data enhances model performance in activity recognition.

Method enables multi-modal reasoning without retraining the entire VLM.

Abstract

Vision-language models (VLMs) have shown powerful capabilities in visual question answering and reasoning tasks by combining visual representations with the abstract skill set large language models (LLMs) learn during pretraining. Vision, while the most popular modality to augment LLMs with, is only one representation of a scene. In human-robot interaction scenarios, robot perception requires accurate scene understanding by the robot. In this paper, we define and demonstrate a method of aligning the embedding spaces of different modalities (in this case, inertial measurement unit (IMU) data) to the vision embedding space through a combination of supervised and contrastive training, enabling the VLM to understand and reason about these additional modalities without retraining. We opt to give the model IMU embeddings directly over using a separate human activity recognition model that feeds directly into the prompt to allow for any nonlinear interactions between the query, image, and IMU signal that would be lost by mapping the IMU data to a discrete activity label. Further, we demonstrate our methodology's efficacy through experiments involving human activity recognition using IMU data and visual inputs. Our results show that using multiple modalities as input improves the VLM's scene understanding and enhances its overall performance in various tasks, thus paving the way for more versatile and capable language models in multi-modal contexts.