Mining Deep And-Or Object Structures via Cost-Sensitive Question-Answer-Based Active Annotations

TL;DR

This paper introduces a cost-sensitive active QA framework for efficiently learning a detailed nine-layer And-Or graph from web images, reducing human supervision and improving object structure understanding.

Contribution

It proposes a novel QA-based active learning method that minimizes overall risk by balancing loss and query costs for hierarchical object structure learning.

Findings

Requires significantly less human supervision

Achieves better performance than baseline methods

Effectively learns detailed object hierarchies from web images

Abstract

This paper presents a cost-sensitive active Question-Answering (QA) framework for learning a nine-layer And-Or graph (AOG) from web images. The AOG explicitly represents object categories, poses/viewpoints, parts, and detailed structures within the parts in a compositional hierarchy. The QA framework is designed to minimize an overall risk, which trades off the loss and query costs. The loss is defined for nodes in all layers of the AOG, including the generative loss (measuring the likelihood of the images) and the discriminative loss (measuring the fitness to human answers). The cost comprises both the human labor of answering questions and the computational cost of model learning. The cost-sensitive QA framework iteratively selects different storylines of questions to update different nodes in the AOG. Experiments showed that our method required much less human supervision (e.g., labeling parts on 3--10 training objects for each category) and achieved better performance than baseline methods.