SIR: Self-supervised Image Rectification via Seeing the Same Scene from Multiple Different Lenses

TL;DR

This paper introduces a self-supervised learning approach for image rectification that leverages multiple distorted views of the same scene to improve generalization and performance without requiring ground-truth distortion parameters.

Contribution

The paper proposes a novel self-supervised framework for image rectification that uses intra- and inter-model consistency across multiple views, eliminating the need for labeled data.

Findings

Achieves comparable or better performance than supervised methods.

Improves the universality and self-consistency of distortion models.

Effective on both synthetic and real-world fisheye images.

Abstract

Deep learning has demonstrated its power in image rectification by leveraging the representation capacity of deep neural networks via supervised training based on a large-scale synthetic dataset. However, the model may overfit the synthetic images and generalize not well on real-world fisheye images due to the limited universality of a specific distortion model and the lack of explicitly modeling the distortion and rectification process. In this paper, we propose a novel self-supervised image rectification (SIR) method based on an important insight that the rectified results of distorted images of a same scene from different lens should be the same. Specifically, we devise a new network architecture with a shared encoder and several prediction heads, each of which predicts the distortion parameter of a specific distortion model. We further leverage a differentiable warping module to generate the rectified images and re-distorted images from the distortion parameters and exploit the intra- and inter-model consistency between them during training, thereby leading to a self-supervised learning scheme without the need for ground-truth distortion parameters or normal images. Experiments on synthetic dataset and real-world fisheye images demonstrate that our method achieves comparable or even better performance than the supervised baseline method and representative state-of-the-art methods. Self-supervised learning also improves the universality of distortion models while keeping their self-consistency.