Does CLIP perceive art the same way we do?

TL;DR

This paper investigates CLIP's ability to perceive and interpret artworks, comparing its understanding of artistic content, style, and context to human perception, revealing both strengths and limitations in its visual representations.

Contribution

It introduces targeted probing tasks to evaluate CLIP's perception of art, highlighting its capabilities and shortcomings in understanding artistic nuances and context.

Findings

CLIP effectively recognizes content and style in artworks.

CLIP shows limitations in understanding artistic intent and aesthetic nuances.

Insights inform better use of CLIP in creative and generative tasks.

Abstract

CLIP has emerged as a powerful multimodal model capable of connecting images and text through joint embeddings, but to what extent does it 'see' the same way humans do - especially when interpreting artworks? In this paper, we investigate CLIP's ability to extract high-level semantic and stylistic information from paintings, including both human-created and AI-generated imagery. We evaluate its perception across multiple dimensions: content, scene understanding, artistic style, historical period, and the presence of visual deformations or artifacts. By designing targeted probing tasks and comparing CLIP's responses to human annotations and expert benchmarks, we explore its alignment with human perceptual and contextual understanding. Our findings reveal both strengths and limitations in CLIP's visual representations, particularly in relation to aesthetic cues and artistic intent. We further discuss the implications of these insights for using CLIP as a guidance mechanism during generative processes, such as style transfer or prompt-based image synthesis. Our work highlights the need for deeper interpretability in multimodal systems, especially when applied to creative domains where nuance and subjectivity play a central role.