Spatio-temporal Vision Transformer for Super-resolution Microscopy

TL;DR

This paper introduces VSR-SIM, a transformer-based method for super-resolution microscopy that enhances temporal resolution in live-cell imaging by effectively capturing motion without traditional motion estimation.

Contribution

The paper presents a novel transformer architecture for SIM reconstruction that handles dynamic samples and improves temporal resolution without relying on optical flow.

Findings

VSR-SIM achieves a ninefold increase in temporal resolution.

The method effectively captures motion in dynamic samples.

VSR-SIM reduces artefacts in SIM reconstructions of live cells.

Abstract

Structured illumination microscopy (SIM) is an optical super-resolution technique that enables live-cell imaging beyond the diffraction limit. Reconstruction of SIM data is prone to artefacts, which becomes problematic when imaging highly dynamic samples because previous methods rely on the assumption that samples are static. We propose a new transformer-based reconstruction method, VSR-SIM, that uses shifted 3-dimensional window multi-head attention in addition to channel attention mechanism to tackle the problem of video super-resolution (VSR) in SIM. The attention mechanisms are found to capture motion in sequences without the need for common motion estimation techniques such as optical flow. We take an approach to training the network that relies solely on simulated data using videos of natural scenery with a model for SIM image formation. We demonstrate a use case enabled by VSR-SIM referred to as rolling SIM imaging, which increases temporal resolution in SIM by a factor of 9. Our method can be applied to any SIM setup enabling precise recordings of dynamic processes in biomedical research with high temporal resolution.