Agile Modeling: From Concept to Classifier in Minutes

TL;DR

This paper introduces Agile Modeling, a real-time, user-in-the-loop approach enabling non-experts to quickly create image classifiers for subjective concepts, overcoming limitations of traditional crowdsourcing.

Contribution

It presents a novel Agile Modeling framework that allows rapid classifier development through minimal user interaction, addressing subjectivity in visual concept labeling.

Findings

Users can create classifiers in under 30 minutes.

The approach outperforms crowdsourcing for subjective concepts.

Simulations show scalability to large datasets like ImageNet21k.

Abstract

The application of computer vision to nuanced subjective use cases is growing. While crowdsourcing has served the vision community well for most objective tasks (such as labeling a "zebra"), it now falters on tasks where there is substantial subjectivity in the concept (such as identifying "gourmet tuna"). However, empowering any user to develop a classifier for their concept is technically difficult: users are neither machine learning experts, nor have the patience to label thousands of examples. In reaction, we introduce the problem of Agile Modeling: the process of turning any subjective visual concept into a computer vision model through a real-time user-in-the-loop interactions. We instantiate an Agile Modeling prototype for image classification and show through a user study (N=14) that users can create classifiers with minimal effort under 30 minutes. We compare this user driven process with the traditional crowdsourcing paradigm and find that the crowd's notion often differs from that of the user's, especially as the concepts become more subjective. Finally, we scale our experiments with simulations of users training classifiers for ImageNet21k categories to further demonstrate the efficacy.