Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning

TL;DR

This paper introduces a deep learning method that virtually refocuses 2D fluorescence microscopy images into 3D, eliminating the need for axial scanning and hardware, while correcting aberrations and enabling cross-modality imaging.

Contribution

It presents a novel deep convolutional neural network framework for 3D virtual refocusing of fluorescence images, significantly enhancing imaging depth and correcting artifacts without additional hardware.

Findings

Achieved 20-fold increase in depth-of-field without hardware or resolution loss.

Successfully corrected sample drift, tilt, and aberrations digitally.

Enabled cross-modality 3D refocusing of fluorescence images.

Abstract

Three-dimensional (3D) fluorescence microscopy in general requires axial scanning to capture images of a sample at different planes. Here we demonstrate that a deep convolutional neural network can be trained to virtually refocus a 2D fluorescence image onto user-defined 3D surfaces within the sample volume. With this data-driven computational microscopy framework, we imaged the neuron activity of a Caenorhabditis elegans worm in 3D using a time-sequence of fluorescence images acquired at a single focal plane, digitally increasing the depth-of-field of the microscope by 20-fold without any axial scanning, additional hardware, or a trade-off of imaging resolution or speed. Furthermore, we demonstrate that this learning-based approach can correct for sample drift, tilt, and other image aberrations, all digitally performed after the acquisition of a single fluorescence image. This unique framework also cross-connects different imaging modalities to each other, enabling 3D refocusing of a single wide-field fluorescence image to match confocal microscopy images acquired at different sample planes. This deep learning-based 3D image refocusing method might be transformative for imaging and tracking of 3D biological samples, especially over extended periods of time, mitigating photo-toxicity, sample drift, aberration and defocusing related challenges associated with standard 3D fluorescence microscopy techniques.