Blocks World Revisited: The Effect of Self-Occlusion on Classification by Convolutional Neural Networks

TL;DR

This paper introduces TEOS, a new 3D dataset designed to study the impact of self-occlusion on object classification by deep neural networks, highlighting the challenge it poses and providing baseline evaluations.

Contribution

The paper presents TEOS, a novel dataset focused on self-occlusion in 3D object classification, and evaluates the performance of existing neural networks on this challenging dataset.

Findings

Self-occlusion significantly affects classification accuracy.

Existing neural networks struggle with TEOS's complexity.

The dataset is publicly available for further research.

Abstract

Despite the recent successes in computer vision, there remain new avenues to explore. In this work, we propose a new dataset to investigate the effect of self-occlusion on deep neural networks. With TEOS (The Effect of Self-Occlusion), we propose a 3D blocks world dataset that focuses on the geometric shape of 3D objects and their omnipresent challenge of self-occlusion. We designed TEOS to investigate the role of self-occlusion in the context of object classification. Even though remarkable progress has been seen in object classification, self-occlusion is a challenge. In the real-world, self-occlusion of 3D objects still presents significant challenges for deep learning approaches. However, humans deal with this by deploying complex strategies, for instance, by changing the viewpoint or manipulating the scene to gather necessary information. With TEOS, we present a dataset of two difficulty levels (L1 and L2 ), containing 36 and 12 objects, respectively. We provide 738 uniformly sampled views of each object, their mask, object and camera position, orientation, amount of self-occlusion, as well as the CAD model of each object. We present baseline evaluations with five well-known classification deep neural networks and show that TEOS poses a significant challenge for all of them. The dataset, as well as the pre-trained models, are made publicly available for the scientific community under https://nvision2.data.eecs.yorku.ca/TEOS.