Probing CLIP's Comprehension of 360-Degree Textual and Visual Semantics

TL;DR

This paper evaluates CLIP's understanding of 360-degree image-text semantics, revealing strengths in textual comprehension but limitations in visual invariance, and proposes fine-tuning methods to improve visual semantic understanding.

Contribution

It introduces novel evaluation methods for 360-degree semantics, analyzes CLIP's capabilities and limitations, and proposes a LoRA-based fine-tuning framework to enhance visual semantic invariance.

Findings

CLIP effectively uses explicit textual identifiers for 360-degree semantics.

CLIP struggles to maintain semantic alignment under horizontal circular shifts.

Fine-tuning improves visual semantic invariance but slightly reduces original semantic performance.

Abstract

The dream of instantly creating rich 360-degree panoramic worlds from text is rapidly becoming a reality, yet a crucial gap exists in our ability to reliably evaluate their semantic alignment. Contrastive Language-Image Pre-training (CLIP) models, standard AI evaluators, predominantly trained on perspective image-text pairs, face an open question regarding their understanding of the unique characteristics of 360-degree panoramic image-text pairs. This paper addresses this gap by first introducing two concepts: \emph{360-degree textual semantics}, semantic information conveyed by explicit format identifiers, and \emph{360-degree visual semantics}, invariant semantics under horizontal circular shifts. To probe CLIP's comprehension of these semantics, we then propose novel evaluation methodologies using keyword manipulation and horizontal circular shifts of varying magnitudes. Rigorous statistical analyses across popular CLIP configurations reveal that: (1) CLIP models effectively leverage explicit textual identifiers, demonstrating an understanding of 360-degree textual semantics; and (2) CLIP models fail to robustly preserve semantic alignment under horizontal circular shifts, indicating limited comprehension of 360-degree visual semantics. To address this limitation, we propose a LoRA-based fine-tuning framework that explicitly instills invariance to circular shifts. Our fine-tuned models exhibit improved comprehension of 360-degree visual semantics, though with a slight degradation in original semantic evaluation performance, highlighting a fundamental trade-off in adapting CLIP to 360-degree panoramic images. Code is available at https://github.com/littlewhitesea/360Semantics.