Cycle-Correspondence Loss: Learning Dense View-Invariant Visual Features from Unlabeled and Unordered RGB Images

TL;DR

This paper introduces Cycle-Correspondence Loss, a self-supervised method for learning dense, view-invariant visual features from unpaired RGB images, simplifying data collection and improving keypoint tracking and robotic grasping performance.

Contribution

The paper proposes a novel cycle-consistency based loss for self-supervised dense descriptor learning that operates on unpaired RGB images, reducing calibration and supervision requirements.

Findings

Outperforms other self-supervised RGB methods in keypoint tracking

Approaches supervised method performance in grasping tasks

Enables simple data collection pipeline for view-invariant features

Abstract

Robot manipulation relying on learned object-centric descriptors became popular in recent years. Visual descriptors can easily describe manipulation task objectives, they can be learned efficiently using self-supervision, and they can encode actuated and even non-rigid objects. However, learning robust, view-invariant keypoints in a self-supervised approach requires a meticulous data collection approach involving precise calibration and expert supervision. In this paper we introduce Cycle-Correspondence Loss (CCL) for view-invariant dense descriptor learning, which adopts the concept of cycle-consistency, enabling a simple data collection pipeline and training on unpaired RGB camera views. The key idea is to autonomously detect valid pixel correspondences by attempting to use a prediction over a new image to predict the original pixel in the original image, while scaling error terms based on the estimated confidence. Our evaluation shows that we outperform other self-supervised RGB-only methods, and approach performance of supervised methods, both with respect to keypoint tracking as well as for a robot grasping downstream task.