Human-Machine CRFs for Identifying Bottlenecks in Holistic Scene Understanding

TL;DR

This paper investigates how integrating human input into different components of CRF models can reveal bottlenecks and guide future research to improve holistic scene understanding in images.

Contribution

It introduces a hybrid human-machine CRF framework to analyze the impact of various tasks on scene understanding performance.

Findings

Identifies key tasks that limit scene understanding improvements

Quantifies potential gains from human intervention in specific components

Provides insights into prioritizing research efforts for better models

Abstract

Recent trends in image understanding have pushed for holistic scene understanding models that jointly reason about various tasks such as object detection, scene recognition, shape analysis, contextual reasoning, and local appearance based classifiers. In this work, we are interested in understanding the roles of these different tasks in improved scene understanding, in particular semantic segmentation, object detection and scene recognition. Towards this goal, we "plug-in" human subjects for each of the various components in a state-of-the-art conditional random field model. Comparisons among various hybrid human-machine CRFs give us indications of how much "head room" there is to improve scene understanding by focusing research efforts on various individual tasks.