Look, Learn and Leverage (L$^3$): Mitigating Visual-Domain Shift and Discovering Intrinsic Relations via Symbolic Alignment

TL;DR

The paper introduces L$^3$, a framework that improves the generalization of models across visual domain shifts by using symbolic alignment with segmentation masks, enabling better transfer of learned relations.

Contribution

L$^3$ systematically decomposes learning into phases and employs symbolic alignment to enhance model robustness against domain shifts and relation absence.

Findings

Outperforms existing methods on DRL, CRL, and VQA tasks

Enables pretrained relation models to be reused effectively across domains

Demonstrates robustness to visual domain shifts and missing relation data

Abstract

Modern deep learning models have demonstrated outstanding performance on discovering the underlying mechanisms when both visual appearance and intrinsic relations (e.g., causal structure) data are sufficient, such as Disentangled Representation Learning (DRL), Causal Representation Learning (CRL) and Visual Question Answering (VQA) methods. However, generalization ability of these models is challenged when the visual domain shifts and the relations data is absent during finetuning. To address this challenge, we propose a novel learning framework, Look, Learn and Leverage (L$^3$), which decomposes the learning process into three distinct phases and systematically utilize the class-agnostic segmentation masks as the common symbolic space to align visual domains. Thus, a relations discovery model can be trained on the source domain, and when the visual domain shifts and the intrinsic relations are absent, the pretrained relations discovery model can be directly reused and maintain a satisfactory performance. Extensive performance evaluations are conducted on three different tasks: DRL, CRL and VQA, and show outstanding results on all three tasks, which reveals the advantages of L$^3$.