Dilated Convolution with Learnable Spacings makes visual models more aligned with humans: a Grad-CAM study

TL;DR

This study demonstrates that Dilated Convolution with Learnable Spacing (DCLS) enhances the interpretability of visual models by aligning their Grad-CAM heatmaps more closely with human visual attention, compared to standard convolutions.

Contribution

The paper shows that replacing standard convolutions with DCLS improves model interpretability and introduces Threshold-Grad-CAM to further enhance heatmap quality across various models.

Findings

DCLS increases interpretability scores in most tested models.

Threshold-Grad-CAM improves heatmap interpretability for models with poor initial results.

DCLS outperforms standard convolutions in aligning model attention with human visual strategies.

Abstract

Dilated Convolution with Learnable Spacing (DCLS) is a recent advanced convolution method that allows enlarging the receptive fields (RF) without increasing the number of parameters, like the dilated convolution, yet without imposing a regular grid. DCLS has been shown to outperform the standard and dilated convolutions on several computer vision benchmarks. Here, we show that, in addition, DCLS increases the models' interpretability, defined as the alignment with human visual strategies. To quantify it, we use the Spearman correlation between the models' GradCAM heatmaps and the ClickMe dataset heatmaps, which reflect human visual attention. We took eight reference models - ResNet50, ConvNeXt (T, S and B), CAFormer, ConvFormer, and FastViT (sa 24 and 36) - and drop-in replaced the standard convolution layers with DCLS ones. This improved the interpretability score in seven of them. Moreover, we observed that Grad-CAM generated random heatmaps for two models in our study: CAFormer and ConvFormer models, leading to low interpretability scores. We addressed this issue by introducing Threshold-Grad-CAM, a modification built on top of Grad-CAM that enhanced interpretability across nearly all models. The code and checkpoints to reproduce this study are available at: https://github.com/rabihchamas/DCLS-GradCAM-Eval.