A Visual Tour Of Current Challenges In Multimodal Language Models

TL;DR

This paper investigates how well multimodal transformer models grounded in imagery can learn function words, revealing limited success and highlighting the need for new datasets and methods to improve their understanding.

Contribution

The study evaluates the capability of stable diffusion multimodal models to learn function words, identifying specific limitations and areas for future research.

Findings

Stable diffusion models effectively model only a small subset of function words.

Models perform better on some pronoun subcategories and relatives.

Significant work remains to improve multimodal learning of function words.

Abstract

Transformer models trained on massive text corpora have become the de facto models for a wide range of natural language processing tasks. However, learning effective word representations for function words remains challenging. Multimodal learning, which visually grounds transformer models in imagery, can overcome the challenges to some extent; however, there is still much work to be done. In this study, we explore the extent to which visual grounding facilitates the acquisition of function words using stable diffusion models that employ multimodal models for text-to-image generation. Out of seven categories of function words, along with numerous subcategories, we find that stable diffusion models effectively model only a small fraction of function words -- a few pronoun subcategories and relatives. We hope that our findings will stimulate the development of new datasets and approaches that enable multimodal models to learn better representations of function words.