Foundation Models Meet Visualizations: Challenges and Opportunities

TL;DR

This paper explores the intersection of foundation models and visualization techniques, discussing challenges and opportunities in using visualizations to understand and enhance foundation models, and vice versa.

Contribution

It introduces a new research paradigm dividing the intersection into visualizations for foundation models and foundation models for visualizations, highlighting challenges and opportunities.

Findings

Visualizations aid in understanding and evaluating foundation models.

Foundation models can enhance visualization techniques.

Identifies key challenges in integrating foundation models with visualization.

Abstract

Recent studies have indicated that foundation models, such as BERT and GPT, excel in adapting to a variety of downstream tasks. This adaptability has established them as the dominant force in building artificial intelligence (AI) systems. As visualization techniques intersect with these models, a new research paradigm emerges. This paper divides these intersections into two main areas: visualizations for foundation models (VIS4FM) and foundation models for visualizations (FM4VIS). In VIS4FM, we explore the primary role of visualizations in understanding, refining, and evaluating these intricate models. This addresses the pressing need for transparency, explainability, fairness, and robustness. Conversely, within FM4VIS, we highlight how foundation models can be utilized to advance the visualization field itself. The confluence of foundation models and visualizations holds great promise, but it also comes with its own set of challenges. By highlighting these challenges and the growing opportunities, this paper seeks to provide a starting point for continued exploration in this promising avenue.