Clean Implicit 3D Structure from Noisy 2D STEM Images

TL;DR

This paper introduces a differentiable model for STEM imaging that jointly learns to denoise 2D images and reconstruct implicit 3D structures without supervision, improving results on synthetic and real data.

Contribution

It proposes a novel differentiable image formation model for STEM that disentangles noise from 3D signals in an implicit 3D reconstruction framework without supervision.

Findings

Successfully disentangles 3D signal and noise

Outperforms several baselines on synthetic data

Effective on real STEM images

Abstract

Scanning Transmission Electron Microscopes (STEMs) acquire 2D images of a 3D sample on the scale of individual cell components. Unfortunately, these 2D images can be too noisy to be fused into a useful 3D structure and facilitating good denoisers is challenging due to the lack of clean-noisy pairs. Additionally, representing a detailed 3D structure can be difficult even for clean data when using regular 3D grids. Addressing these two limitations, we suggest a differentiable image formation model for STEM, allowing to learn a joint model of 2D sensor noise in STEM together with an implicit 3D model. We show, that the combination of these models are able to successfully disentangle 3D signal and noise without supervision and outperform at the same time several baselines on synthetic and real data.