Lightweight Optimal-Transport Harmonization on Edge Devices

TL;DR

This paper introduces a lightweight, real-time color harmonization method for augmented reality on edge devices, leveraging optimal transport theory and a compact encoder to improve composite image quality.

Contribution

It proposes a novel, efficient approach using classical optimal transport with a trained encoder for on-device color harmonization in AR.

Findings

Achieves superior scores on AR composite images compared to state-of-the-art methods.

Supports real-time inference on edge devices.

Provides a new dataset and toolkit for further research.

Abstract

Color harmonization adjusts the colors of an inserted object so that it perceptually matches the surrounding image, resulting in a seamless composite. The harmonization problem naturally arises in augmented reality (AR), yet harmonization algorithms are not currently integrated into AR pipelines because real-time solutions are scarce. In this work, we address color harmonization for AR by proposing a lightweight approach that supports on-device inference. For this, we leverage classical optimal transport theory by training a compact encoder to predict the Monge-Kantorovich transport map. We benchmark our MKL-Harmonizer algorithm against state-of-the-art methods and demonstrate that for real composite AR images our method achieves the best aggregated score. We release our dedicated AR dataset of composite images with pixel-accurate masks and data-gathering toolkit to support further data acquisition by researchers.