Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metric

TL;DR

This paper introduces TopoLip, a novel metric combining topological data analysis and Lipschitz continuity to evaluate and compare the robustness of attention and convolution models.

Contribution

The paper presents TopoLip, a unified framework for theoretical and empirical robustness assessment applicable across different neural network architectures.

Findings

Attention models show smoother transformations and higher robustness.

TopoLip effectively correlates model architecture with robustness.

The framework enables direct comparison of model robustness.

Abstract

Robustness is a critical aspect of machine learning models. Existing robustness evaluation approaches often lack theoretical generality or rely heavily on empirical assessments, limiting insights into the structural factors contributing to robustness. Moreover, theoretical robustness analysis is not applicable for direct comparisons between models. To address these challenges, we propose $\textit{TopoLip}$, a metric based on layer-wise analysis that bridges topological data analysis and Lipschitz continuity for robustness evaluation. TopoLip provides a unified framework for both theoretical and empirical robustness comparisons across different architectures or configurations, and it reveals how model parameters influence the robustness of models. Using TopoLip, we demonstrate that attention-based models typically exhibit smoother transformations and greater robustness compared to convolution-based models, as validated through theoretical analysis and adversarial tasks. Our findings establish a connection between architectural design, robustness, and topological properties.