Treatment Learning Causal Transformer for Noisy Image Classification

TL;DR

This paper introduces a causal transformer model that improves noisy image classification by estimating the effect of noise as a treatment, leveraging causal inference to enhance robustness against various noise types.

Contribution

It proposes the Treatment Learning Causal Transformer (TLT), a novel architecture that incorporates causal treatment effects into image classification, addressing noise robustness.

Findings

TLT outperforms existing models on noisy datasets.

The model effectively estimates noise treatment effects.

Improves visual salience methods for noisy images.

Abstract

Current top-notch deep learning (DL) based vision models are primarily based on exploring and exploiting the inherent correlations between training data samples and their associated labels. However, a known practical challenge is their degraded performance against "noisy" data, induced by different circumstances such as spurious correlations, irrelevant contexts, domain shift, and adversarial attacks. In this work, we incorporate this binary information of "existence of noise" as treatment into image classification tasks to improve prediction accuracy by jointly estimating their treatment effects. Motivated from causal variational inference, we propose a transformer-based architecture, Treatment Learning Causal Transformer (TLT), that uses a latent generative model to estimate robust feature representations from current observational input for noise image classification. Depending on the estimated noise level (modeled as a binary treatment factor), TLT assigns the corresponding inference network trained by the designed causal loss for prediction. We also create new noisy image datasets incorporating a wide range of noise factors (e.g., object masking, style transfer, and adversarial perturbation) for performance benchmarking. The superior performance of TLT in noisy image classification is further validated by several refutation evaluation metrics. As a by-product, TLT also improves visual salience methods for perceiving noisy images.