RiCS: A 2D Self-Occlusion Map for Harmonizing Volumetric Objects

TL;DR

RiCS introduces a novel 2D self-occlusion map derived from 3D data using ray-marching in camera space, improving human image harmonization and segmentation by effectively modeling occlusions and shadows.

Contribution

The paper proposes RiCS, a new 2D self-occlusion representation from 3D data that enhances image harmonization and segmentation tasks involving complex occlusions and shadows.

Findings

Improved image quality in human harmonization tasks.

Enhanced modeling of complex shadow effects.

Significant performance boost in segmentation networks.

Abstract

There have been remarkable successes in computer vision with deep learning. While such breakthroughs show robust performance, there have still been many challenges in learning in-depth knowledge, like occlusion or predicting physical interactions. Although some recent works show the potential of 3D data in serving such context, it is unclear how we efficiently provide 3D input to the 2D models due to the misalignment in dimensionality between 2D and 3D. To leverage the successes of 2D models in predicting self-occlusions, we design Ray-marching in Camera Space (RiCS), a new method to represent the self-occlusions of foreground objects in 3D into a 2D self-occlusion map. We test the effectiveness of our representation on the human image harmonization task by predicting shading that is coherent with a given background image. Our experiments demonstrate that our representation map not only allows us to enhance the image quality but also to model temporally coherent complex shadow effects compared with the simulation-to-real and harmonization methods, both quantitatively and qualitatively. We further show that we can significantly improve the performance of human parts segmentation networks trained on existing synthetic datasets by enhancing the harmonization quality with our method.