Leveraging Depth Maps and Attention Mechanisms for Enhanced Image Inpainting

TL;DR

This paper introduces a dual-encoder deep learning model that combines RGB images and depth maps with attention mechanisms to improve image inpainting accuracy, especially under occlusions, outperforming baseline methods.

Contribution

The novel integration of depth maps with RGB images using a dual encoder and attention mechanism significantly enhances inpainting quality over traditional RGB-only approaches.

Findings

Depth information improves reconstruction accuracy.

Attention mechanisms further boost inpainting performance.

Model outperforms baseline methods in qualitative and quantitative tests.

Abstract

Existing deep learning-based image inpainting methods typically rely on convolutional networks with RGB images to reconstruct images. However, relying exclusively on RGB images may neglect important depth information, which plays a critical role in understanding the spatial and structural context of a scene. Just as human vision leverages stereo cues to perceive depth, incorporating depth maps into the inpainting process can enhance the model's ability to reconstruct images with greater accuracy and contextual awareness. In this paper, we propose a novel approach that incorporates both RGB and depth images for enhanced image inpainting. Our models employ a dual encoder architecture, where one encoder processes the RGB image and the other handles the depth image. The encoded features from both encoders are then fused in the decoder using an attention mechanism, effectively integrating the RGB and depth representations. We use two different masking strategies, line and square, to test the robustness of the model under different types of occlusions. To further analyze the effectiveness of our approach, we use Gradient-weighted Class Activation Mapping (Grad-CAM) visualizations to examine the regions of interest the model focuses on during inpainting. We show that incorporating depth information alongside the RGB image significantly improves the reconstruction quality. Through both qualitative and quantitative comparisons, we demonstrate that the depth-integrated model outperforms the baseline, with attention mechanisms further enhancing inpainting performance, as evidenced by multiple evaluation metrics and visualization.