Modeling Selective Feature Attention for Representation-based Siamese Text Matching

TL;DR

This paper introduces Feature Attention and Selective Feature Attention mechanisms for Siamese text matching networks, enhancing feature dependency modeling and semantic extraction, leading to improved performance across benchmarks.

Contribution

The paper proposes novel Feature Attention and Selective Feature Attention modules that dynamically emphasize important features and enable multi-scale semantic extraction in Siamese networks.

Findings

Feature Attention improves dependency modeling among features.

Selective Feature Attention enhances semantic extraction across abstraction levels.

The proposed modules outperform baseline models on multiple benchmarks.

Abstract

Representation-based Siamese networks have risen to popularity in lightweight text matching due to their low deployment and inference costs. While word-level attention mechanisms have been implemented within Siamese networks to improve performance, we propose Feature Attention (FA), a novel downstream block designed to enrich the modeling of dependencies among embedding features. Employing "squeeze-and-excitation" techniques, the FA block dynamically adjusts the emphasis on individual features, enabling the network to concentrate more on features that significantly contribute to the final classification. Building upon FA, we introduce a dynamic "selection" mechanism called Selective Feature Attention (SFA), which leverages a stacked BiGRU Inception structure. The SFA block facilitates multi-scale semantic extraction by traversing different stacked BiGRU layers, encouraging the network to selectively concentrate on semantic information and embedding features across varying levels of abstraction. Both the FA and SFA blocks offer a seamless integration capability with various Siamese networks, showcasing a plug-and-play characteristic. Experimental evaluations conducted across diverse text matching baselines and benchmarks underscore the indispensability of modeling feature attention and the superiority of the "selection" mechanism.