MARMOT: Masked Autoencoder for Modeling Transient Imaging

TL;DR

MARMOT is a self-supervised Transformer-based autoencoder trained on large NLOS transient datasets, enabling improved modeling and reconstruction of hidden objects in transient imaging applications.

Contribution

The paper introduces MARMOT, a novel masked autoencoder that leverages self-supervised learning on large datasets for modeling transient imaging in NLOS scenarios.

Findings

MARMOT outperforms state-of-the-art methods in NLOS transient imaging.

Pretraining on large datasets enhances downstream task performance.

MARMOT effectively predicts full transient measurements from partial data.

Abstract

Pretrained models have demonstrated impressive success in many modalities such as language and vision. Recent works facilitate the pretraining paradigm in imaging research. Transients are a novel modality, which are captured for an object as photon counts versus arrival times using a precisely time-resolved sensor. In particular for non-line-of-sight (NLOS) scenarios, transients of hidden objects are measured beyond the sensor's direct line of sight. Using NLOS transients, the majority of previous works optimize volume density or surfaces to reconstruct the hidden objects and do not transfer priors learned from datasets. In this work, we present a masked autoencoder for modeling transient imaging, or MARMOT, to facilitate NLOS applications. Our MARMOT is a self-supervised model pretrianed on massive and diverse NLOS transient datasets. Using a Transformer-based encoder-decoder, MARMOT learns features from partially masked transients via a scanning pattern mask (SPM), where the unmasked subset is functionally equivalent to arbitrary sampling, and predicts full measurements. Pretrained on TransVerse-a synthesized transient dataset of 500K 3D models-MARMOT adapts to downstream imaging tasks using direct feature transfer or decoder finetuning. Comprehensive experiments are carried out in comparisons with state-of-the-art methods. Quantitative and qualitative results demonstrate the efficiency of our MARMOT.