SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding

TL;DR

SYNAPSE is a training-free framework that systematically analyzes and stress-tests Transformer models' internal neuron representations across domains, revealing stability, redundancies, and vulnerabilities without retraining.

Contribution

It introduces a novel, systematic, training-free method for neuron-level interpretability and robustness analysis applicable across different Transformer architectures and domains.

Findings

Internal representations are domain-independent and redundantly encode task information.

Class-specific sensitivities reveal heterogeneous neuron specialization.

Small manipulations can redirect model predictions, exposing vulnerabilities.

Abstract

In recent years, Artificial Intelligence has become a powerful partner for complex tasks such as data analysis, prediction, and problem-solving, yet its lack of transparency raises concerns about its reliability. In sensitive domains such as healthcare or cybersecurity, ensuring transparency, trustworthiness, and robustness is essential, since the consequences of wrong decisions or successful attacks can be severe. Prior neuron-level interpretability approaches are primarily descriptive, task-dependent, or require retraining, which limits their use as systematic, reusable tools for evaluating internal robustness across architectures and domains. To overcome these limitations, this work proposes SYNAPSE, a systematic, training-free framework for understanding and stress-testing the internal behavior of Transformer models across domains. It extracts per-layer [CLS] representations, trains a lightweight linear probe to obtain global and per-class neuron rankings, and applies forward-hook interventions during inference. This design enables controlled experiments on internal representations without altering the original model, thereby allowing weaknesses, stability patterns, and label-specific sensitivities to be measured and compared directly across tasks and architectures. Across all experiments, SYNAPSE reveals a consistent, domain-independent organization of internal representations, in which task-relevant information is encoded in broad, overlapping neuron subsets. This redundancy provides a strong degree of functional stability, while class-wise asymmetries expose heterogeneous specialization patterns and enable label-aware analysis. In contrast, small structured manipulations in weight or logit space are sufficient to redirect predictions, highlighting complementary vulnerability profiles and illustrating how SYNAPSE can guide the development of more robust Transformer models.