SAGE: Saliency-Guided Contrastive Embeddings

TL;DR

SAGE introduces a novel contrastive loss that leverages human saliency in the model's latent space, improving neural network generalization and robustness by guiding focus towards salient features during training.

Contribution

The paper proposes SAGE, a new loss function that uses human saliency and contrastive embeddings in latent space, moving beyond image space guidance for better model training.

Findings

SAGE improves classification accuracy over state-of-the-art saliency methods.

It demonstrates robustness across different neural network architectures.

Effective in both open- and closed-set scenarios.

Abstract

Integrating human perceptual priors into the training of neural networks has been shown to raise model generalization, serve as an effective regularizer, and align models with human expertise for applications in high-risk domains. Existing approaches to integrate saliency into model training often rely on internal model mechanisms, which recent research suggests may be unreliable. Our insight is that many challenges associated with saliency-guided training stem from the placement of the guidance approaches solely within the image space. Instead, we move away from the image space, use the model's latent space embeddings to steer human guidance during training, and we propose SAGE (Saliency-Guided Contrastive Embeddings): a loss function that integrates human saliency into network training using contrastive embeddings. We apply salient-preserving and saliency-degrading signal augmentations to the input and capture the changes in embeddings and model logits. We guide the model towards salient features and away from non-salient features using a contrastive triplet loss. Additionally, we perform a sanity check on the logit distributions to ensure that the model outputs match the saliency-based augmentations. We demonstrate a boost in classification performance across both open- and closed-set scenarios against SOTA saliency-based methods, showing SAGE's effectiveness across various backbones, and include experiments to suggest its wide generalization across tasks.