FocDepthFormer: Transformer with latent LSTM for Depth Estimation from Focal Stack

TL;DR

FocDepthFormer introduces a Transformer-LSTM hybrid model for depth estimation from focal stacks, enabling flexible stack length processing and improved accuracy over existing CNN-based methods.

Contribution

The paper proposes a novel Transformer-based network with an LSTM module for depth estimation from focal stacks, overcoming fixed stack length limitations of prior CNN methods.

Findings

Outperforms state-of-the-art methods on multiple benchmarks

Can be pre-trained on monocular RGB datasets to enhance performance

Effectively processes focal stacks of arbitrary length

Abstract

Most existing methods for depth estimation from a focal stack of images employ convolutional neural networks (CNNs) using 2D or 3D convolutions over a fixed set of images. However, their effectiveness is constrained by the local properties of CNN kernels, which restricts them to process only focal stacks of fixed number of images during both training and inference. This limitation hampers their ability to generalize to stacks of arbitrary lengths. To overcome these limitations, we present a novel Transformer-based network, FocDepthFormer, which integrates a Transformer with an LSTM module and a CNN decoder. The Transformer's self-attention mechanism allows for the learning of more informative spatial features by implicitly performing non-local cross-referencing. The LSTM module is designed to integrate representations across image stacks of varying lengths. Additionally, we employ multi-scale convolutional kernels in an early-stage encoder to capture low-level features at different degrees of focus/defocus. By incorporating the LSTM, FocDepthFormer can be pre-trained on large-scale monocular RGB depth estimation datasets, improving visual pattern learning and reducing reliance on difficult-to-obtain focal stack data. Extensive experiments on diverse focal stack benchmark datasets demonstrate that our model outperforms state-of-the-art approaches across multiple evaluation metrics.