Agile Deliberation: Concept Deliberation for Subjective Visual Classification

TL;DR

This paper introduces Agile Deliberation, a human-in-the-loop framework that enables users to iteratively refine subjective visual concepts through structured deliberation stages, improving classifier alignment and user understanding.

Contribution

It operationalizes concept deliberation into a two-stage framework supporting subjective, evolving concepts, validated through extensive user studies rather than standard benchmarks.

Findings

Achieved 7.5% higher F1 scores than automated baselines.

Surpassed manual deliberation by over 3% in F1 score.

Participants reported clearer understanding and lower cognitive effort.

Abstract

From content moderation to content curation, applications requiring vision classifiers for visual concepts are rapidly expanding. Existing human-in-the-loop approaches typically assume users begin with a clear, stable concept understanding to be able to provide high-quality supervision. In reality, users often start with a vague idea and must iteratively refine it through "concept deliberation", a practice we uncovered through structured interviews with content moderation experts. We operationalize the common strategies in deliberation used by real content moderators into a human-in-the-loop framework called "Agile Deliberation" that explicitly supports evolving and subjective concepts. The system supports users in defining the concept for themselves by exposing them to borderline cases. The system does this with two deliberation stages: (1) concept scoping, which decomposes the initial concept into a structured hierarchy of sub-concepts, and (2) concept iteration, which surfaces semantically borderline examples for user reflection and feedback to iteratively align an image classifier with the user's evolving intent. Since concept deliberation is inherently subjective and interactive, we painstakingly evaluate the framework through 18 user sessions, each 1.5h long, rather than standard benchmarking datasets. We find that Agile Deliberation achieves 7.5% higher F1 scores than automated decomposition baselines and more than 3% higher than manual deliberation, while participants reported clearer conceptual understanding and lower cognitive effort.