Textual Training for the Hassle-Free Removal of Unwanted Visual Data: Case Studies on OOD and Hateful Image Detection

TL;DR

This paper introduces Hassle-Free Textual Training (HFTT), a novel method that leverages synthetic textual data and pre-trained vision-language models to detect unwanted visual content without extensive human annotation.

Contribution

We propose HFTT, a new streamlined approach that uses only textual data and pre-trained models for detecting unwanted visual data, reducing annotation effort and extending to abstract concept detection.

Findings

HFTT effectively detects out-of-distribution and hateful images.

HFTT reduces human annotation needs significantly.

The method performs well across multiple detection tasks.

Abstract

In our study, we explore methods for detecting unwanted content lurking in visual datasets. We provide a theoretical analysis demonstrating that a model capable of successfully partitioning visual data can be obtained using only textual data. Based on the analysis, we propose Hassle-Free Textual Training (HFTT), a streamlined method capable of acquiring detectors for unwanted visual content, using only synthetic textual data in conjunction with pre-trained vision-language models. HFTT features an innovative objective function that significantly reduces the necessity for human involvement in data annotation. Furthermore, HFTT employs a clever textual data synthesis method, effectively emulating the integration of unknown visual data distribution into the training process at no extra cost. The unique characteristics of HFTT extend its utility beyond traditional out-of-distribution detection, making it applicable to tasks that address more abstract concepts. We complement our analyses with experiments in out-of-distribution detection and hateful image detection. Our codes are available at https://github.com/Saehyung-Lee/HFTT