LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences

TL;DR

LOTUS is a comprehensive leaderboard for detailed image captioning that evaluates quality, societal biases, and user preferences, revealing diverse model strengths and the importance of tailored assessments.

Contribution

Introduces LOTUS, a new evaluation framework addressing gaps in captioning assessment by incorporating bias and user preference considerations.

Findings

No single model excels across all criteria

Caption detail correlates with bias risks

User preferences influence optimal model choice

Abstract

Large Vision-Language Models (LVLMs) have transformed image captioning, shifting from concise captions to detailed descriptions. We introduce LOTUS, a leaderboard for evaluating detailed captions, addressing three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. LOTUS comprehensively evaluates various aspects, including caption quality (e.g., alignment, descriptiveness), risks (\eg, hallucination), and societal biases (e.g., gender bias) while enabling preference-oriented evaluations by tailoring criteria to diverse user preferences. Our analysis of recent LVLMs reveals no single model excels across all criteria, while correlations emerge between caption detail and bias risks. Preference-oriented evaluations demonstrate that optimal model selection depends on user priorities.