Learning to Answer Multilingual and Code-Mixed Questions

TL;DR

This paper advances multilingual and code-mixed question-answering by proposing new techniques that achieve state-of-the-art results across multiple domains, addressing the challenge of limited high-quality multilingual datasets.

Contribution

It introduces novel methods for multilingual and code-mixed QA, and tackles multi-hop question generation using multiple documents, improving performance across various tasks.

Findings

Achieved state-of-the-art performance on answer extraction, ranking, and generation.

Developed generic techniques applicable across multiple domains and languages.

Enhanced QA systems to better handle multilingual and code-mixed queries.

Abstract

Question-answering (QA) that comes naturally to humans is a critical component in seamless human-computer interaction. It has emerged as one of the most convenient and natural methods to interact with the web and is especially desirable in voice-controlled environments. Despite being one of the oldest research areas, the current QA system faces the critical challenge of handling multilingual queries. To build an Artificial Intelligent (AI) agent that can serve multilingual end users, a QA system is required to be language versatile and tailored to suit the multilingual environment. Recent advances in QA models have enabled surpassing human performance primarily due to the availability of a sizable amount of high-quality datasets. However, the majority of such annotated datasets are expensive to create and are only confined to the English language, making it challenging to acknowledge progress in foreign languages. Therefore, to measure a similar improvement in the multilingual QA system, it is necessary to invest in high-quality multilingual evaluation benchmarks. In this dissertation, we focus on advancing QA techniques for handling end-user queries in multilingual environments. This dissertation consists of two parts. In the first part, we explore multilingualism and a new dimension of multilingualism referred to as code-mixing. Second, we propose a technique to solve the task of multi-hop question generation by exploiting multiple documents. Experiments show our models achieve state-of-the-art performance on answer extraction, ranking, and generation tasks on multiple domains of MQA, VQA, and language generation. The proposed techniques are generic and can be widely used in various domains and languages to advance QA systems.