Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models

TL;DR

This study examines how different visual text styles in images influence Large Visual Language Models' attribute-based descriptions of concepts, revealing style leakage affecting semantic inference.

Contribution

It uncovers the impact of visual text styles on LVLMs' semantic descriptions and highlights the need for style-aware evaluation and mitigation strategies.

Findings

Text style influences LVLMs' attribute-based descriptions.

Style leakage affects semantic inference even when concepts are correctly identified.

Results motivate style-aware evaluation and mitigation for multimedia systems.

Abstract

When the visual style of text is considered, a wide variety can be observed in font, color, and size. However, when a word is read, its meaning is independent of the style in which it has been written or rendered. In this paper, we investigate whether, and how, the style in which a word is visualized in an image impacts the description that a Large Visual Language Model (LVLM) provides for the concept to which that word refers. Specifically, we investigate how functional text styles (readability-oriented, e.g., black sans-serif) versus decorative styles (display-oriented, e.g., colored cursive/script) affect LVLMs' descriptions of a concept in terms of the attributes of that concept. Our experiments study the situation in which the LVLM is able to correctly identify the concept referred to by a visual text, i.e., by a word or words rendered as an image, and in which the visual text style should not influence the attribute-based description that the LVLM produces. Our experimental results reveal that even when the concept is correctly identified, text style influences the model's attribute-based descriptions of the concept. Our findings demonstrate non-trivial style leakage from text style into semantic inference and motivate style-aware evaluation and mitigation for LVLM-based multimedia systems.