Beyond topography: Topographic regularization improves robustness and reshapes representations in convolutional neural networks

TL;DR

This paper investigates how topographic regularization in convolutional neural networks influences robustness and internal representations, demonstrating that different regularization methods systematically reshape network structure and improve resilience to perturbations.

Contribution

It compares two novel topographic regularization methods, Weight Similarity and Activation Similarity, showing their distinct effects on robustness and representational organization in CNNs.

Findings

Both regularizations improve robustness to input and weight perturbations.

WS produces smooth topographies with correlated neighborhoods.

AS results in bimodal correlation structures lacking spatial smoothness.

Abstract

Topographic convolutional neural networks (TCNNs) are computational models that can simulate aspects of the brain's spatial and functional organization. However, it is unclear whether and how different types of topographic regularization shape robustness, representational structure, and functional organization during end-to-end training. We address this question by comparing TCNNs trained with two local spatial losses applied to a penultimate-layer topographic grid: i) Weight Similarity (WS), whose objective penalizes differences between neighboring units' incoming weight vectors, and ii) Activation Similarity (AS), whose objective penalizes differences between neighboring units' activation patterns over stimuli. We evaluate the trained models on classification accuracy, robustness to weight perturbations and input degradation, the spatial organization of learned representations, and development of category-selective "expert units" in the penultimate layer. Both losses changed inter-unit correlation structure, but in qualitatively different ways. WS produced smooth topographies, with correlated neighborhoods. In contrast, AS produced a bimodal inter-unit correlation structure that lacked spatial smoothness. AS and WS training increased robustness relative to control (non-topographic) models: AS improved robustness to image degradation on CIFAR-10, WS did so on MNIST, and both improved robustness to weight perturbations. WS was also associated with greater input sensitivity at the unit level and stronger functional localization. In addition, as compared to control models, both AS and WS produced differences in orientation tuning, symmetry sensitivity, and eccentricity profiles of units. Together, these results show that local topographic regularization can improve robustness during end-to-end training while systematically reshaping representational structure.