SILT: Efficient transformer training for inter-lingual inference

TL;DR

This paper introduces SILT, a novel efficient transformer architecture that aligns multilingual embeddings for inter-lingual Natural Language Inference, reducing parameters and achieving state-of-the-art results across benchmarks.

Contribution

SILT is a new architecture that enables efficient inter-lingual NLI by leveraging frozen multilingual transformers and a matrix alignment method, reducing training complexity.

Findings

SILT achieves state-of-the-art performance on multilingual NLI benchmarks.

The model drastically reduces the number of trainable parameters.

SILT effectively processes unmatched language pairs in inter-lingual inference.

Abstract

The ability of transformers to perform precision tasks such as question answering, Natural Language Inference (NLI) or summarising, have enabled them to be ranked as one of the best paradigm to address Natural Language Processing (NLP) tasks. NLI is one of the best scenarios to test these architectures, due to the knowledge required to understand complex sentences and established relationships between a hypothesis and a premise. Nevertheless, these models suffer from incapacity to generalise to other domains or difficulties to face multilingual and interlingual scenarios. The leading pathway in the literature to address these issues involve designing and training extremely large architectures, which leads to unpredictable behaviours and to establish barriers which impede broad access and fine tuning. In this paper, we propose a new architecture called Siamese Inter-Lingual Transformer (SILT), to efficiently align multilingual embeddings for Natural Language Inference, allowing for unmatched language pairs to be processed. SILT leverages siamese pre-trained multi-lingual transformers with frozen weights where the two input sentences attend each other to later be combined through a matrix alignment method. The experimental results carried out in this paper evidence that SILT allows to reduce drastically the number of trainable parameters while allowing for inter-lingual NLI and achieving state-of-the-art performance on common benchmarks. We make our code and dataset available at https://github.com/jahuerta92/siamese-inter-lingual-transformer.