UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory

TL;DR

UberNet is a unified CNN architecture capable of performing multiple low-, mid-, and high-level vision tasks simultaneously, trained efficiently on diverse datasets with limited memory, achieving competitive results in real-time.

Contribution

This work introduces UberNet, a novel end-to-end trainable CNN that handles a wide range of vision tasks within a single model, addressing training on diverse datasets and memory constraints.

Findings

Handles multiple vision tasks simultaneously

Achieves real-time performance (~0.7 seconds per frame)

Maintains competitive accuracy across tasks

Abstract

In this work we introduce a convolutional neural network (CNN) that jointly handles low-, mid-, and high-level vision tasks in a unified architecture that is trained end-to-end. Such a universal network can act like a `swiss knife' for vision tasks; we call this architecture an UberNet to indicate its overarching nature. We address two main technical challenges that emerge when broadening up the range of tasks handled by a single CNN: (i) training a deep architecture while relying on diverse training sets and (ii) training many (potentially unlimited) tasks with a limited memory budget. Properly addressing these two problems allows us to train accurate predictors for a host of tasks, without compromising accuracy. Through these advances we train in an end-to-end manner a CNN that simultaneously addresses (a) boundary detection (b) normal estimation (c) saliency estimation (d) semantic segmentation (e) human part segmentation (f) semantic boundary detection, (g) region proposal generation and object detection. We obtain competitive performance while jointly addressing all of these tasks in 0.7 seconds per frame on a single GPU. A demonstration of this system can be found at http://cvn.ecp.fr/ubernet/.