RIPE: Reinforcement Learning on Unlabeled Image Pairs for Robust Keypoint Extraction

TL;DR

RIPE is a reinforcement learning framework that trains a robust keypoint extractor using minimal supervision from image pairs, achieving competitive results without relying on extensive labeled data or 3D models.

Contribution

It introduces a weakly-supervised reinforcement learning approach for keypoint extraction that reduces data annotation requirements and enhances robustness.

Findings

Achieves competitive performance on standard benchmarks.

Simplifies data preparation for keypoint extraction.

Utilizes intermediate encoder layers for improved description.

Abstract

We introduce RIPE, an innovative reinforcement learning-based framework for weakly-supervised training of a keypoint extractor that excels in both detection and description tasks. In contrast to conventional training regimes that depend heavily on artificial transformations, pre-generated models, or 3D data, RIPE requires only a binary label indicating whether paired images represent the same scene. This minimal supervision significantly expands the pool of training data, enabling the creation of a highly generalized and robust keypoint extractor. RIPE utilizes the encoder's intermediate layers for the description of the keypoints with a hyper-column approach to integrate information from different scales. Additionally, we propose an auxiliary loss to enhance the discriminative capability of the learned descriptors. Comprehensive evaluations on standard benchmarks demonstrate that RIPE simplifies data preparation while achieving competitive performance compared to state-of-the-art techniques, marking a significant advancement in robust keypoint extraction and description. To support further research, we have made our code publicly available at https://github.com/fraunhoferhhi/RIPE.