Comics for Everyone: Generating Accessible Text Descriptions for Comic Strips

TL;DR

This paper presents a method to generate accessible text descriptions for comic strips by combining computer vision and large language models, aiming to improve accessibility for the blind or low vision community.

Contribution

It introduces a novel two-step approach that integrates computer vision with multimodal language models to produce detailed comic descriptions for accessibility.

Findings

Encouraging performance on annotated comic datasets

Effective extraction of panel, character, and text information

Promising results in both quantitative and qualitative evaluations

Abstract

Comic strips are a popular and expressive form of visual storytelling that can convey humor, emotion, and information. However, they are inaccessible to the BLV (Blind or Low Vision) community, who cannot perceive the images, layouts, and text of comics. Our goal in this paper is to create natural language descriptions of comic strips that are accessible to the visually impaired community. Our method consists of two steps: first, we use computer vision techniques to extract information about the panels, characters, and text of the comic images; second, we use this information as additional context to prompt a multimodal large language model (MLLM) to produce the descriptions. We test our method on a collection of comics that have been annotated by human experts and measure its performance using both quantitative and qualitative metrics. The outcomes of our experiments are encouraging and promising.