NeuralMarker: A Framework for Learning General Marker Correspondence

TL;DR

NeuralMarker introduces a neural network framework for estimating dense marker correspondences from general markers like posters to images, outperforming traditional methods especially under challenging conditions, and enabling applications like AR and video editing.

Contribution

The paper presents NeuralMarker, a novel neural network-based framework for dense marker correspondence estimation that handles complex conditions and introduces a new benchmark for evaluation.

Findings

NeuralMarker significantly outperforms previous correspondence methods.

The framework effectively handles marker deformation and harsh lighting conditions.

It enables new applications in AR and video editing.

Abstract

We tackle the problem of estimating correspondences from a general marker, such as a movie poster, to an image that captures such a marker. Conventionally, this problem is addressed by fitting a homography model based on sparse feature matching. However, they are only able to handle plane-like markers and the sparse features do not sufficiently utilize appearance information. In this paper, we propose a novel framework NeuralMarker, training a neural network estimating dense marker correspondences under various challenging conditions, such as marker deformation, harsh lighting, etc. Besides, we also propose a novel marker correspondence evaluation method circumstancing annotations on real marker-image pairs and create a new benchmark. We show that NeuralMarker significantly outperforms previous methods and enables new interesting applications, including Augmented Reality (AR) and video editing.