Integration of multi-level semantics in PTMs with an attention model for question matching

TL;DR

This paper introduces a new attention-based model for question matching that improves performance by integrating different levels of semantics from pre-trained language models.

Contribution

The novel ERNIE-ATT model integrates multi-level semantics from PTMs to enhance question matching robustness.

Findings

ERNIE-ATT outperforms traditional models and strong PTM-based models on challenging datasets.

The model demonstrates improved robustness in complex question matching scenarios.

Abstract

The task of question matching/retrieval focuses on determining whether two questions are semantically equivalent. It has garnered significant attention in the field of natural language processing (NLP) due to its commercial value. While neural network models have made great strides and achieved human-level accuracy, they still face challenges when handling complex scenarios. In this paper, we delve into the utilization of different specializations encoded in different layers of large-scale pre-trained language models (PTMs). We propose a novel attention-based model called ERNIE-ATT that effectively integrates the diverse levels of semantics acquired by PTMs, thereby enhancing robustness. Experimental evaluations on two challenging datasets showcase the superior performance of our proposed model. It outperforms not only traditional models that do not use PTMs but also exhibits a significant improvement over strong PTM-based models. These findings demonstrate the effectiveness of our approach in enhancing the robustness of question matching/retrieval systems.

Figures