SAP: Syntactic Attention Pruning for Transformer-based Language Models

TL;DR

SAP introduces a syntactic and attention pattern-based head pruning method for Transformer models, improving interpretability and robustness while maintaining performance without retraining.

Contribution

The paper proposes Syntactic Attention Pruning (SAP), a novel approach that leverages syntactic structures and attention patterns for more effective Transformer head pruning.

Findings

SAP outperforms existing pruning methods in retrain-free settings.

SAP preserves critical attention heads with strong attention values.

Candidate Filtering enhances robustness during pruning.

Abstract

This paper introduces Syntactic Attention Pruning (SAP), a novel method for effectively pruning attention heads in Transformer models. Unlike conventional approaches that rely solely on mathematical analysis of model weights and activations, SAP incorporates both the syntactic structure and attention patterns of sentences to guide the pruning process. By leveraging these linguistic features, SAP not only achieves performance comparable to state-of-the-art methods but also enhances the interpretability of model behavior. To further improve robustness, we propose Candidate Filtering (CF), a mechanism that prioritizes heads based on their contribution to model performance, mitigating degradation during pruning. Experimental results indicate that SAP effectively preserves critical heads of a high density of strong attention values, outperforming existing head pruning strategies in retrain-free settings. These findings position SAP as a promising foundation for a new direction in model compression research, offering high flexibility for pruning across all transformer-based language models.