Probing Human Visual Robustness with Neurally-Guided Deep Neural Networks

TL;DR

This study demonstrates that aligning deep neural networks with human neural responses across the ventral visual stream enhances their robustness to image perturbations, highlighting the importance of hierarchical neural representations and manifold properties.

Contribution

The paper introduces a neurally-guided training approach that aligns DNN representations with human VVS responses, leading to hierarchical robustness improvements and insights into neural manifold geometry.

Findings

Hierarchical alignment with VVS improves DNN robustness.

Neural category manifolds become smaller and more separable across VVS.

Neural manifold guidance alone can enhance robustness.

Abstract

Humans effortlessly navigate the dynamic visual world, yet deep neural networks (DNNs), despite excelling at many visual tasks, are surprisingly vulnerable to minor image perturbations. Past theories suggest that human visual robustness arises from a representational space that evolves along the ventral visual stream (VVS) of the brain to increasingly tolerate object transformations. To test whether robustness is supported by such progression as opposed to being confined exclusively to specialized higher-order regions, we trained DNNs to align their representations with human neural responses from consecutive VVS regions while performing visual tasks. We demonstrate a hierarchical improvement in DNN robustness: alignment to higher-order VVS regions leads to greater improvement. To investigate the mechanism behind such robustness gains, we test a prominent hypothesis that attributes human robustness to the unique geometry of neural category manifolds in the VVS. We first reveal that more desirable manifold properties, specifically, smaller extent and better linear separability, indeed emerge across the human VVS. These properties can be inherited by neurally aligned DNNs and predict their subsequent robustness gains. Furthermore, we show that supervision from neural manifolds alone, via manifold guidance, is sufficient to qualitatively reproduce the hierarchical robustness improvements. Together, these results highlight the critical role of the evolving representational space across VVS in achieving robust visual inference, in part through the formation of more linearly separable category manifolds, which may in turn be leveraged to develop more robust AI systems.