The Cross-Depiction Problem: Computer Vision Algorithms for Recognising Objects in Artwork and in Photographs

TL;DR

This paper investigates the challenge of recognizing objects across different depiction styles like photographs and artwork, benchmarking various methods and highlighting the importance of spatial relations in improving robustness.

Contribution

It provides a comprehensive benchmark of classification, domain adaptation, and deep learning methods for the cross-depiction problem, revealing their limitations and emphasizing the role of spatial relations.

Findings

Deep learning methods perform poorly on diverse depiction styles.

Models with strong spatial relation understanding are more robust.

Most methods struggle with inhomogeneous databases.

Abstract

The cross-depiction problem is that of recognising visual objects regardless of whether they are photographed, painted, drawn, etc. It is a potentially significant yet under-researched problem. Emulating the remarkable human ability to recognise objects in an astonishingly wide variety of depictive forms is likely to advance both the foundations and the applications of Computer Vision. In this paper we benchmark classification, domain adaptation, and deep learning methods; demonstrating that none perform consistently well in the cross-depiction problem. Given the current interest in deep learning, the fact such methods exhibit the same behaviour as all but one other method: they show a significant fall in performance over inhomogeneous databases compared to their peak performance, which is always over data comprising photographs only. Rather, we find the methods that have strong models of spatial relations between parts tend to be more robust and therefore conclude that such information is important in modelling object classes regardless of appearance details.