Collaboration among Image and Object Level Features for Image Colourisation

TL;DR

This paper introduces UCapsNet, a self-supervised neural network that combines image-level and object-level features via capsules and skip connections to improve image colourisation quality without human input.

Contribution

The work presents a novel single network architecture, UCapsNet, integrating CNNs and capsule networks for object-aware colourisation in a fully self-supervised manner.

Findings

Outperforms existing methods on standard quality metrics

Achieves state-of-the-art results in image colourisation

Preferred by users in large-scale studies

Abstract

Image colourisation is an ill-posed problem, with multiple correct solutions which depend on the context and object instances present in the input datum. Previous approaches attacked the problem either by requiring intense user interactions or by exploiting the ability of convolutional neural networks (CNNs) in learning image level (context) features. However, obtaining human hints is not always feasible and CNNs alone are not able to learn object-level semantics unless multiple models pretrained with supervision are considered. In this work, we propose a single network, named UCapsNet, that separate image-level features obtained through convolutions and object-level features captured by means of capsules. Then, by skip connections over different layers, we enforce collaboration between such disentangling factors to produce high quality and plausible image colourisation. We pose the problem as a classification task that can be addressed by a fully self-supervised approach, thus requires no human effort. Experimental results on three benchmark datasets show that our approach outperforms existing methods on standard quality metrics and achieves a state of the art performances on image colourisation. A large scale user study shows that our method is preferred over existing solutions.