Blind Inpainting of Large-scale Masks of Thin Structures with Adversarial and Reinforcement Learning

TL;DR

This paper introduces a novel neural network-based method for blind inpainting of large-scale thin structures in segmentation masks, utilizing adversarial training and reinforcement learning to ensure global coherence and local detail preservation across diverse imaging domains.

Contribution

It presents a new approach combining adversarial and reinforcement learning to effectively inpaint large-scale thin structures without known missing regions, maintaining global structure and topology.

Findings

Outperforms existing methods in producing realistic inpainted results

Achieves results comparable to human performance in plant root inpainting

Demonstrates effectiveness across medical, plant science, and remote sensing datasets

Abstract

Several imaging applications (vessels, retina, plant roots, road networks from satellites) require the accurate segmentation of thin structures for subsequent analysis. Discontinuities (gaps) in the extracted foreground may hinder down-stream image-based analysis of biomarkers, organ structure and topology. In this paper, we propose a general post-processing technique to recover such gaps in large-scale segmentation masks. We cast this problem as a blind inpainting task, where the regions of missing lines in the segmentation masks are not known to the algorithm, which we solve with an adversarially trained neural network. One challenge of using large images is the memory capacity of current GPUs. The typical approach of dividing a large image into smaller patches to train the network does not guarantee global coherence of the reconstructed image that preserves structure and topology. We use adversarial training and reinforcement learning (Policy Gradient) to endow the model with both global context and local details. We evaluate our method in several datasets in medical imaging, plant science, and remote sensing. Our experiments demonstrate that our model produces the most realistic and complete inpainted results, outperforming other approaches. In a dedicated study on plant roots we find that our approach is also comparable to human performance. Implementation available at \url{https://github.com/Hhhhhhhhhhao/Thin-Structure-Inpainting}.