GCISG: Guided Causal Invariant Learning for Improved Syn-to-real Generalization

TL;DR

This paper introduces a causal invariance learning approach to improve the generalization of models trained on synthetic data when applied to real-world tasks, addressing domain gaps caused by style differences.

Contribution

It proposes a novel causal framework and a style-invariant feature learning method to enhance synthetic-to-real generalization in visual tasks.

Findings

Achieves state-of-the-art results on image classification.

Improves semantic segmentation performance.

Effectively reduces domain gap impact.

Abstract

Training a deep learning model with artificially generated data can be an alternative when training data are scarce, yet it suffers from poor generalization performance due to a large domain gap. In this paper, we characterize the domain gap by using a causal framework for data generation. We assume that the real and synthetic data have common content variables but different style variables. Thus, a model trained on synthetic dataset might have poor generalization as the model learns the nuisance style variables. To that end, we propose causal invariance learning which encourages the model to learn a style-invariant representation that enhances the syn-to-real generalization. Furthermore, we propose a simple yet effective feature distillation method that prevents catastrophic forgetting of semantic knowledge of the real domain. In sum, we refer to our method as Guided Causal Invariant Syn-to-real Generalization that effectively improves the performance of syn-to-real generalization. We empirically verify the validity of proposed methods, and especially, our method achieves state-of-the-art on visual syn-to-real domain generalization tasks such as image classification and semantic segmentation.