Comateformer: Combined Attention Transformer for Semantic Sentence Matching

TL;DR

Comateformer introduces a novel transformer-based quasi-attention mechanism that enhances semantic sentence matching by capturing subtle differences and dissimilarities, leading to improved performance on multiple datasets.

Contribution

The paper proposes a new quasi-attention mechanism within transformers that models both similarity and dissimilarity for better semantic matching.

Findings

Achieves consistent improvements across ten datasets.

Effectively captures subtle differences between sentences.

Demonstrates robustness in various testing scenarios.

Abstract

The Transformer-based model have made significant strides in semantic matching tasks by capturing connections between phrase pairs. However, to assess the relevance of sentence pairs, it is insufficient to just examine the general similarity between the sentences. It is crucial to also consider the tiny subtleties that differentiate them from each other. Regrettably, attention softmax operations in transformers tend to miss these subtle differences. To this end, in this work, we propose a novel semantic sentence matching model named Combined Attention Network based on Transformer model (Comateformer). In Comateformer model, we design a novel transformer-based quasi-attention mechanism with compositional properties. Unlike traditional attention mechanisms that merely adjust the weights of input tokens, our proposed method learns how to combine, subtract, or resize specific vectors when building a representation. Moreover, our proposed approach builds on the intuition of similarity and dissimilarity (negative affinity) when calculating dual affinity scores. This allows for a more meaningful representation of relationships between sentences. To evaluate the performance of our proposed model, we conducted extensive experiments on ten public real-world datasets and robustness testing. Experimental results show that our method achieves consistent improvements.