ELITE: Enhanced Language-Image Toxicity Evaluation for Safety
Wonjun Lee, Doehyeon Lee, Eugene Choi, Sangyoon Yu, Ashkan Yousefpour, Haon Park, Bumsub Ham, Suhyun Kim

TL;DR
The paper introduces ELITE, a new safety benchmark and evaluation method for Vision Language Models, improving detection of harmful content and diversity in safety assessments to promote safer AI applications.
Contribution
It presents the ELITE benchmark and evaluator, which enhance safety evaluation by filtering ambiguous data and incorporating explicit toxicity scoring for multimodal content.
Findings
ELITE evaluator aligns better with human judgments than previous methods.
ELITE benchmark offers higher quality and diversity in safety evaluation data.
The approach improves detection of implicit and explicit harmful content.
Abstract
Current Vision Language Models (VLMs) remain vulnerable to malicious prompts that induce harmful outputs. Existing safety benchmarks for VLMs primarily rely on automated evaluation methods, but these methods struggle to detect implicit harmful content or produce inaccurate evaluations. Therefore, we found that existing benchmarks have low levels of harmfulness, ambiguous data, and limited diversity in image-text pair combinations. To address these issues, we propose the ELITE benchmark, a high-quality safety evaluation benchmark for VLMs, underpinned by our enhanced evaluation method, the ELITE evaluator. The ELITE evaluator explicitly incorporates a toxicity score to accurately assess harmfulness in multimodal contexts, where VLMs often provide specific, convincing, but unharmful descriptions of images. We filter out ambiguous and low-quality image-text pairs from existing benchmarks…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInfrastructure Maintenance and Monitoring
