Neural Image Representations for Multi-Image Fusion and Layer Separation

TL;DR

This paper introduces a neural image representation framework for aligning and fusing multiple images into a single view, effectively handling camera motion and scene changes, and enabling layer separation tasks.

Contribution

The proposed framework uniquely combines neural image representations with various alignment strategies for multi-image fusion and layer separation without relying on a reference frame.

Findings

Effective multi-image fusion across different scene motions

Successful layer separation using neural representations

No need for reference image selection

Abstract

We propose a framework for aligning and fusing multiple images into a single view using neural image representations (NIRs), also known as implicit or coordinate-based neural representations. Our framework targets burst images that exhibit camera ego motion and potential changes in the scene. We describe different strategies for alignment depending on the nature of the scene motion -- namely, perspective planar (i.e., homography), optical flow with minimal scene change, and optical flow with notable occlusion and disocclusion. With the neural image representation, our framework effectively combines multiple inputs into a single canonical view without the need for selecting one of the images as a reference frame. We demonstrate how to use this multi-frame fusion framework for various layer separation tasks. The code and results are available at https://shnnam.github.io/research/nir.