Multi-scale Self-calibrated Network for Image Light Source Transfer

TL;DR

This paper introduces a multi-scale self-calibrated network for image light source transfer, improving feature representation and semantic understanding in relighting tasks, leading to better performance on the VIDIT dataset.

Contribution

It proposes novel self-calibrated blocks for feature encoding and decoding, and fuses multi-scale features to enhance scene reconstruction and relighting accuracy.

Findings

Significant performance improvement on VIDIT dataset

Effective calibration of feature representations

Enhanced semantic information utilization

Abstract

Image light source transfer (LLST), as the most challenging task in the domain of image relighting, has attracted extensive attention in recent years. In the latest research, LLST is decomposed three sub-tasks: scene reconversion, shadow estimation, and image re-rendering, which provides a new paradigm for image relighting. However, many problems for scene reconversion and shadow estimation tasks, including uncalibrated feature information and poor semantic information, are still unresolved, thereby resulting in insufficient feature representation. In this paper, we propose novel down-sampling feature self-calibrated block (DFSB) and up-sampling feature self-calibrated block (UFSB) as the basic blocks of feature encoder and decoder to calibrate feature representation iteratively because the LLST is similar to the recalibration of image light source. In addition, we fuse the multi-scale features of the decoder in scene reconversion task to further explore and exploit more semantic information, thereby providing more accurate primary scene structure for image re-rendering. Experimental results in the VIDIT dataset show that the proposed approach significantly improves the performance for LLST.