DeepFormableTag: End-to-end Generation and Recognition of Deformable Fiducial Markers

TL;DR

This paper introduces a novel end-to-end system for generating and recognizing deformable fiducial markers that are robust to shape deformation, optical distortions, and motion blur, enabling high-accuracy detection in challenging conditions.

Contribution

The authors propose a joint optimization of a deformable marker generator and detector using a differentiable photorealistic renderer, advancing marker recognition robustness and versatility.

Findings

Successfully decodes 36-bit messages at ~29 fps under severe deformation

Outperforms traditional and data-driven marker methods in accuracy

Enables new applications like motion capture and active 3D scanning

Abstract

Fiducial markers have been broadly used to identify objects or embed messages that can be detected by a camera. Primarily, existing detection methods assume that markers are printed on ideally planar surfaces. Markers often fail to be recognized due to various imaging artifacts of optical/perspective distortion and motion blur. To overcome these limitations, we propose a novel deformable fiducial marker system that consists of three main parts: First, a fiducial marker generator creates a set of free-form color patterns to encode significantly large-scale information in unique visual codes. Second, a differentiable image simulator creates a training dataset of photorealistic scene images with the deformed markers, being rendered during optimization in a differentiable manner. The rendered images include realistic shading with specular reflection, optical distortion, defocus and motion blur, color alteration, imaging noise, and shape deformation of markers. Lastly, a trained marker detector seeks the regions of interest and recognizes multiple marker patterns simultaneously via inverse deformation transformation. The deformable marker creator and detector networks are jointly optimized via the differentiable photorealistic renderer in an end-to-end manner, allowing us to robustly recognize a wide range of deformable markers with high accuracy. Our deformable marker system is capable of decoding 36-bit messages successfully at ~29 fps with severe shape deformation. Results validate that our system significantly outperforms the traditional and data-driven marker methods. Our learning-based marker system opens up new interesting applications of fiducial markers, including cost-effective motion capture of the human body, active 3D scanning using our fiducial markers' array as structured light patterns, and robust augmented reality rendering of virtual objects on dynamic surfaces.