Learning Geometry-aware Representations by Sketching

TL;DR

This paper introduces Learning by Sketching (LBS), a method that converts images into sketches with geometric information, improving scene understanding and downstream vision tasks without requiring sketch datasets.

Contribution

LBS is a novel approach that learns to generate geometric sketches from images in a single step, preserving geometric information without needing explicit sketch supervision.

Findings

Improves object attribute classification on CLEVR dataset

Enhances domain transfer between CLEVR and STL-10 datasets

Provides rich geometric information for various downstream tasks

Abstract

Understanding geometric concepts, such as distance and shape, is essential for understanding the real world and also for many vision tasks. To incorporate such information into a visual representation of a scene, we propose learning to represent the scene by sketching, inspired by human behavior. Our method, coined Learning by Sketching (LBS), learns to convert an image into a set of colored strokes that explicitly incorporate the geometric information of the scene in a single inference step without requiring a sketch dataset. A sketch is then generated from the strokes where CLIP-based perceptual loss maintains a semantic similarity between the sketch and the image. We show theoretically that sketching is equivariant with respect to arbitrary affine transformations and thus provably preserves geometric information. Experimental results show that LBS substantially improves the performance of object attribute classification on the unlabeled CLEVR dataset, domain transfer between CLEVR and STL-10 datasets, and for diverse downstream tasks, confirming that LBS provides rich geometric information.