Captioning Visualizations with Large Language Models (CVLLM): A Tutorial

TL;DR

This tutorial explores how large language models can be used to automatically generate captions for visualizations, highlighting recent advances, applications, and future directions in the field.

Contribution

It provides a comprehensive overview of applying large language models to visualization captioning, including neural models, transformer architectures, and emerging research directions.

Findings

LLMs enable improved visualization captioning.

Transformer architectures are central to recent advances.

Future research directions are identified for further development.

Abstract

Automatically captioning visualizations is not new, but recent advances in large language models(LLMs) open exciting new possibilities. In this tutorial, after providing a brief review of Information Visualization (InfoVis) principles and past work in captioning, we introduce neural models and the transformer architecture used in generic LLMs. We then discuss their recent applications in InfoVis, with a focus on captioning. Additionally, we explore promising future directions in this field.