Discriminative Training of Deep Fully-connected Continuous CRF with Task-specific Loss

TL;DR

This paper introduces a fully-connected deep continuous CRF model that jointly learns unary and pairwise potentials with CNNs, optimized with task-specific loss functions for improved structured prediction in vision tasks.

Contribution

It presents a novel deep continuous CRF framework with end-to-end training and task-specific loss functions, enhancing performance on both discrete and continuous vision problems.

Findings

Outperforms state-of-the-art on semantic labeling

Achieves robust depth estimation results

Effectively uses task-specific loss for improved MAP inference

Abstract

Recent works on deep conditional random fields (CRF) have set new records on many vision tasks involving structured predictions. Here we propose a fully-connected deep continuous CRF model for both discrete and continuous labelling problems. We exemplify the usefulness of the proposed model on multi-class semantic labelling (discrete) and the robust depth estimation (continuous) problems. In our framework, we model both the unary and the pairwise potential functions as deep convolutional neural networks (CNN), which are jointly learned in an end-to-end fashion. The proposed method possesses the main advantage of continuously-valued CRF, which is a closed-form solution for the Maximum a posteriori (MAP) inference. To better adapt to different tasks, instead of using the commonly employed maximum likelihood CRF parameter learning protocol, we propose task-specific loss functions for learning the CRF parameters. It enables direct optimization of the quality of the MAP estimates during the course of learning. Specifically, we optimize the multi-class classification loss for the semantic labelling task and the Turkey's biweight loss for the robust depth estimation problem. Experimental results on the semantic labelling and robust depth estimation tasks demonstrate that the proposed method compare favorably against both baseline and state-of-the-art methods. In particular, we show that although the proposed deep CRF model is continuously valued, with the equipment of task-specific loss, it achieves impressive results even on discrete labelling tasks.