Foundation Models Meet Visualizations: Challenges and Opportunities
Weikai Yang, Mengchen Liu, Zheng Wang, and Shixia Liu

TL;DR
This paper explores the intersection of foundation models and visualization techniques, discussing challenges and opportunities in using visualizations to understand and enhance foundation models, and vice versa.
Contribution
It introduces a new research paradigm dividing the intersection into visualizations for foundation models and foundation models for visualizations, highlighting challenges and opportunities.
Findings
Visualizations aid in understanding and evaluating foundation models.
Foundation models can enhance visualization techniques.
Identifies key challenges in integrating foundation models with visualization.
Abstract
Recent studies have indicated that foundation models, such as BERT and GPT, excel in adapting to a variety of downstream tasks. This adaptability has established them as the dominant force in building artificial intelligence (AI) systems. As visualization techniques intersect with these models, a new research paradigm emerges. This paper divides these intersections into two main areas: visualizations for foundation models (VIS4FM) and foundation models for visualizations (FM4VIS). In VIS4FM, we explore the primary role of visualizations in understanding, refining, and evaluating these intricate models. This addresses the pressing need for transparency, explainability, fairness, and robustness. Conversely, within FM4VIS, we highlight how foundation models can be utilized to advance the visualization field itself. The confluence of foundation models and visualizations holds great promise,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsScientific Computing and Data Management · Explainable Artificial Intelligence (XAI) · Ethics and Social Impacts of AI
MethodsMulti-Head Attention · Attention Is All You Need · Linear Layer · Cosine Annealing · Discriminative Fine-Tuning · Dropout · Weight Decay · Softmax · Byte Pair Encoding · Linear Warmup With Cosine Annealing
