Injecting Relational Structural Representation in Neural Networks for Question Similarity

TL;DR

This paper introduces a method to incorporate syntactic structural information into neural networks for question similarity by leveraging Tree Kernels and large-scale pre-training, improving accuracy on benchmark datasets.

Contribution

It proposes a novel approach combining Tree Kernel-based SVMs and neural network pre-training to better utilize syntactic structures in question similarity tasks.

Findings

Improved accuracy on Quora and SemEval datasets.

Effective use of Tree Kernels for structural representation.

Enhanced neural network performance after fine-tuning.

Abstract

Effectively using full syntactic parsing information in Neural Networks (NNs) to solve relational tasks, e.g., question similarity, is still an open problem. In this paper, we propose to inject structural representations in NNs by (i) learning an SVM model using Tree Kernels (TKs) on relatively few pairs of questions (few thousands) as gold standard (GS) training data is typically scarce, (ii) predicting labels on a very large corpus of question pairs, and (iii) pre-training NNs on such large corpus. The results on Quora and SemEval question similarity datasets show that NNs trained with our approach can learn more accurate models, especially after fine tuning on GS.