Disharmony: Forensics using Reverse Lighting Harmonization

TL;DR

This paper introduces Disharmony, a forensic model that leverages reverse lighting harmonization data and segmentation to improve detection of manipulated or generated images, especially harmonized objects.

Contribution

The study presents a novel Disharmony Network that effectively detects harmonized and edited image regions by utilizing an aggregated dataset of harmonization techniques.

Findings

Outperforms existing forensic models in detecting harmonized objects

Effective in identifying various image edits including virtual try-on

Addresses gap in detecting harmonized objects relative to backgrounds

Abstract

Content generation and manipulation approaches based on deep learning methods have seen significant advancements, leading to an increased need for techniques to detect whether an image has been generated or edited. Another area of research focuses on the insertion and harmonization of objects within images. In this study, we explore the potential of using harmonization data in conjunction with a segmentation model to enhance the detection of edited image regions. These edits can be either manually crafted or generated using deep learning methods. Our findings demonstrate that this approach can effectively identify such edits. Existing forensic models often overlook the detection of harmonized objects in relation to the background, but our proposed Disharmony Network addresses this gap. By utilizing an aggregated dataset of harmonization techniques, our model outperforms existing forensic networks in identifying harmonized objects integrated into their backgrounds, and shows potential for detecting various forms of edits, including virtual try-on tasks.