FedQAS: Privacy-aware machine reading comprehension with federated learning

TL;DR

FedQAS introduces a federated learning-based system for machine reading comprehension that preserves privacy while enabling large-scale training on private datasets, utilizing transformer models and federated frameworks.

Contribution

It presents a novel privacy-aware MRC system combining federated learning and transformer models, allowing secure knowledge sharing without central data pooling.

Findings

Successfully deployed as a proof-of-concept with FEDn framework

Achieved competitive performance on SQuAD dataset

Demonstrated privacy preservation in federated MRC setting

Abstract

Machine reading comprehension (MRC) of text data is one important task in Natural Language Understanding. It is a complex NLP problem with a lot of ongoing research fueled by the release of the Stanford Question Answering Dataset (SQuAD) and Conversational Question Answering (CoQA). It is considered to be an effort to teach computers how to "understand" a text, and then to be able to answer questions about it using deep learning. However, until now large-scale training on private text data and knowledge sharing has been missing for this NLP task. Hence, we present FedQAS, a privacy-preserving machine reading system capable of leveraging large-scale private data without the need to pool those datasets in a central location. The proposed approach combines transformer models and federated learning technologies. The system is developed using the FEDn framework and deployed as a proof-of-concept alliance initiative. FedQAS is flexible, language-agnostic, and allows intuitive participation and execution of local model training. In addition, we present the architecture and implementation of the system, as well as provide a reference evaluation based on the SQUAD dataset, to showcase how it overcomes data privacy issues and enables knowledge sharing between alliance members in a Federated learning setting.