SSL Framework for Causal Inconsistency between Structures and Representations

TL;DR

This paper introduces a self-supervised learning framework to address causal inconsistency in complex data types, enhancing causal structure and representation learning in deep learning models.

Contribution

It identifies causal inconsistency in Indefinite Data and proposes a novel SSL framework based on intervention to improve causal alignment.

Findings

SSL framework improves causal consistency

Enhances causal structure and representation learning

Benefits extend to downstream tasks and LLM instructions

Abstract

The cross-pollination between causal discovery and deep learning has led to increasingly extensive interactions. It results in a large number of deep learning data types (such as images, text, etc.) extending into the field of causal discovery, and a multitude of deep learning tasks have begun to utilize causal discovery to explore the internal causal structure and causal representation of data. In this paper, we first identified that a complex data type, ``Indefinite Data", has conflicts between causal relationships expressed by the causal structure and causal representation generated by deep learning models, a phenomenon referred to as causal inconsistency. We thoroughly analyzed related work to explain why only Indefinite Data exhibits causal inconsistency while other data types do not. Furthermore, to alleviate causal inconsistency, we proposed a self-supervised learning (SSL) framework based on intervention, hoping to provide more causal information from different intervention views to promote consistency between structure and representation. Extensive experiments have shown that the SSL framework enhances causal consistency and can further improve causal structure and representation learning performance. Additionally, we extended the SSL framework to three different downstream tasks and LLM instructions. The quantitative results of these applications all reflect the performance improvement brought about by causal consistency.