Which private attributes do VLMs agree on and predict well?

TL;DR

This paper evaluates open-source Visual Language Models for privacy attribute detection, revealing their tendencies to agree with humans on certain attributes and their potential to enhance privacy annotation processes.

Contribution

It provides a zero-shot evaluation of VLMs for privacy attributes, identifying where they agree with humans and how they can complement human annotations.

Findings

VLMs often predict privacy attributes more frequently than humans.

High agreement among VLMs can help identify overlooked privacy attributes.

VLMs can support large-scale privacy annotation efforts.

Abstract

Visual Language Models (VLMs) are often used for zero-shot detection of visual attributes in the image. We present a zero-shot evaluation of open-source VLMs for privacy-related attribute recognition. We identify the attributes for which VLMs exhibit strong inter-annotator agreement, and discuss the disagreement cases of human and VLM annotations. Our results show that when evaluated against human annotations, VLMs tend to predict the presence of privacy attributes more often than human annotators. In addition to this, we find that in cases of high inter-annotator agreement between VLMs, they can complement human annotation by identifying attributes overlooked by human annotators. This highlights the potential of VLMs to support privacy annotations in large-scale image datasets.