An Experimental Study of Semantic Continuity for Deep Learning Models

TL;DR

This paper investigates semantic discontinuity in deep learning models caused by training targets, proposing a semantic continuity constraint that improves robustness, interpretability, and reduces bias.

Contribution

It introduces a simple semantic continuity constraint that enhances model semantic understanding and robustness, supported by theoretical analysis and experimental validation.

Findings

Semantic continuity constraint reduces non-semantic information use

Improves adversarial robustness and interpretability

Enhances model transfer and reduces bias

Abstract

Deep learning models suffer from the problem of semantic discontinuity: small perturbations in the input space tend to cause semantic-level interference to the model output. We argue that the semantic discontinuity results from these inappropriate training targets and contributes to notorious issues such as adversarial robustness, interpretability, etc. We first conduct data analysis to provide evidence of semantic discontinuity in existing deep learning models, and then design a simple semantic continuity constraint which theoretically enables models to obtain smooth gradients and learn semantic-oriented features. Qualitative and quantitative experiments prove that semantically continuous models successfully reduce the use of non-semantic information, which further contributes to the improvement in adversarial robustness, interpretability, model transfer, and machine bias.