Towards Holistic Scene Understanding: Feedback Enabled Cascaded Classification Models

TL;DR

This paper introduces Feedback Enabled Cascaded Classification Models (FE-CCM), a method that jointly optimizes multiple scene understanding sub-tasks using a feedback mechanism, improving accuracy without altering individual classifiers.

Contribution

The paper presents a novel feedback-enabled cascade approach that enhances multi-task scene understanding by leveraging correlated outputs without modifying existing classifiers.

Findings

Significant performance improvements across all sub-tasks.

Effective in robotic applications like object grasping and finding.

Operates with only black-box access to classifiers.

Abstract

Scene understanding includes many related sub-tasks, such as scene categorization, depth estimation, object detection, etc. Each of these sub-tasks is often notoriously hard, and state-of-the-art classifiers already exist for many of them. These classifiers operate on the same raw image and provide correlated outputs. It is desirable to have an algorithm that can capture such correlation without requiring any changes to the inner workings of any classifier. We propose Feedback Enabled Cascaded Classification Models (FE-CCM), that jointly optimizes all the sub-tasks, while requiring only a `black-box' interface to the original classifier for each sub-task. We use a two-layer cascade of classifiers, which are repeated instantiations of the original ones, with the output of the first layer fed into the second layer as input. Our training method involves a feedback step that allows later classifiers to provide earlier classifiers information about which error modes to focus on. We show that our method significantly improves performance in all the sub-tasks in the domain of scene understanding, where we consider depth estimation, scene categorization, event categorization, object detection, geometric labeling and saliency detection. Our method also improves performance in two robotic applications: an object-grasping robot and an object-finding robot.