MetaPix: Domain Transfer for Semantic Segmentation by Meta Pixel Weighting

TL;DR

MetaPix introduces a meta-learning approach to optimize pixel-level weighting of synthetic data for semantic segmentation, outperforming complex existing methods by directly minimizing target loss.

Contribution

We propose a novel meta-learning method for pixel-level synthetic data weighting that improves semantic segmentation performance without complex heuristics.

Findings

Outperforms combined existing methods with a single meta module

Effective pixel-level weighting learned via gradient-on-gradient technique

Achieves superior segmentation accuracy on benchmark datasets

Abstract

Training a deep neural model for semantic segmentation requires collecting a large amount of pixel-level labeled data. To alleviate the data scarcity problem presented in the real world, one could utilize synthetic data whose label is easy to obtain. Previous work has shown that the performance of a semantic segmentation model can be improved by training jointly with real and synthetic examples with a proper weighting on the synthetic data. Such weighting was learned by a heuristic to maximize the similarity between synthetic and real examples. In our work, we instead learn a pixel-level weighting of the synthetic data by meta-learning, i.e., the learning of weighting should only be minimizing the loss on the target task. We achieve this by gradient-on-gradient technique to propagate the target loss back into the parameters of the weighting model. The experiments show that our method with only one single meta module can outperform a complicated combination of an adversarial feature alignment, a reconstruction loss, plus a hierarchical heuristic weighting at pixel, region and image levels.