Identifying Corresponding Patches in SAR and Optical Images with a Pseudo-Siamese CNN

TL;DR

This paper introduces a pseudo-siamese CNN architecture designed to identify matching patches in high-resolution SAR and optical images, demonstrating high accuracy in complex urban environments and showing potential for multi-sensor image matching.

Contribution

The paper presents a novel pseudo-siamese CNN model for cross-sensor patch matching in SAR and optical imagery, trained on a new dataset with complex urban scenes.

Findings

High accuracy in matching SAR and optical patches

Effective on complex urban scenes with elevated objects

Potential for generalized multi-sensor key-point matching

Abstract

In this letter, we propose a pseudo-siamese convolutional neural network (CNN) architecture that enables to solve the task of identifying corresponding patches in very-high-resolution (VHR) optical and synthetic aperture radar (SAR) remote sensing imagery. Using eight convolutional layers each in two parallel network streams, a fully connected layer for the fusion of the features learned in each stream, and a loss function based on binary cross-entropy, we achieve a one-hot indication if two patches correspond or not. The network is trained and tested on an automatically generated dataset that is based on a deterministic alignment of SAR and optical imagery via previously reconstructed and subsequently co-registered 3D point clouds. The satellite images, from which the patches comprising our dataset are extracted, show a complex urban scene containing many elevated objects (i.e. buildings), thus providing one of the most difficult experimental environments. The achieved results show that the network is able to predict corresponding patches with high accuracy, thus indicating great potential for further development towards a generalized multi-sensor key-point matching procedure. Index Terms-synthetic aperture radar (SAR), optical imagery, data fusion, deep learning, convolutional neural networks (CNN), image matching, deep matching