LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models

TL;DR

LlavaGuard is an open framework utilizing vision-language models to enhance safety and moderation in large-scale visual datasets and models, offering customizable safeguards and high accuracy.

Contribution

The paper introduces LlavaGuard, a comprehensive, open-source VLM-based safety framework with a new safety dataset, advanced augmentation techniques, and versatile models for safety evaluation.

Findings

LlavaGuard outperforms existing safeguards in accuracy.

Models effectively handle diverse safety policies.

Demonstrated success in dataset annotation and content moderation.

Abstract

This paper introduces LlavaGuard, a suite of VLM-based vision safeguards that address the critical need for reliable guardrails in the era of large-scale data and models. To this end, we establish a novel open framework, describing a customizable safety taxonomy, data preprocessing, augmentation, and training setup. For teaching a VLM safeguard on safety, we further create a multimodal safety dataset with high-quality human expert annotations, where each image is labeled with a safety rating, category, and rationale. We also employ advanced augmentations to support context-specific assessments. The resulting LlavaGuard models, ranging from 0.5B to 7B, serve as a versatile tool for evaluating the safety compliance of visual content against flexible policies. In comprehensive experiments, LlavaGuard outperforms both state-of-the-art safeguards and VLMs in accuracy and in flexibly handling different policies. Additionally, we demonstrate LlavaGuard's performance in two real-world applications: large-scale dataset annotation and moderation of text-to-image models. We make our entire framework, including the dataset, model weights, and training code.