A new dataset and comparison for multi-camera frame synthesis

TL;DR

This paper introduces a novel multi-camera dataset to fairly compare frame interpolation and view synthesis methods, revealing that deep learning does not always outperform classical techniques on real data but can excel in synthetic scenes.

Contribution

The paper presents a new multi-camera dataset enabling direct comparison of frame interpolation and view synthesis methods, and evaluates classical and deep learning approaches on this dataset.

Findings

Deep learning methods do not significantly outperform classical methods on real images.

3D Gaussian Splatting underperforms frame interpolators by up to 3.5 dB PSNR on real data.

In synthetic scenes, 3D Gaussian Splatting outperforms frame interpolation by nearly 5 dB PSNR.

Abstract

Many methods exist for frame synthesis in image sequences but can be broadly categorised into frame interpolation and view synthesis techniques. Fundamentally, both frame interpolation and view synthesis tackle the same task, interpolating a frame given surrounding frames in time or space. However, most frame interpolation datasets focus on temporal aspects with single cameras moving through time and space, while view synthesis datasets are typically biased toward stereoscopic depth estimation use cases. This makes direct comparison between view synthesis and frame interpolation methods challenging. In this paper, we develop a novel multi-camera dataset using a custom-built dense linear camera array to enable fair comparison between these approaches. We evaluate classical and deep learning frame interpolators against a view synthesis method (3D Gaussian Splatting) for the task of view in-betweening. Our results reveal that deep learning methods do not significantly outperform classical methods on real image data, with 3D Gaussian Splatting actually underperforming frame interpolators by as much as 3.5 dB PSNR. However, in synthetic scenes, the situation reverses -- 3D Gaussian Splatting outperforms frame interpolation algorithms by almost 5 dB PSNR at a 95% confidence level.