Ensemble Learning-Based Approach for Improving Generalization Capability of Machine Reading Comprehension Systems

TL;DR

This paper explores ensemble learning techniques to enhance the generalization and out-of-distribution accuracy of Machine Reading Comprehension systems without retraining large models, demonstrating improved robustness across datasets.

Contribution

It introduces ensemble methods applied to pre-trained MRC models, analyzing their effectiveness and robustness in out-of-distribution scenarios without additional training.

Findings

Ensemble methods improve out-of-distribution accuracy.

Heterogeneous and hybrid ensembles are most effective.

Ensembles increase robustness against data shifts.

Abstract

Machine Reading Comprehension (MRC) is an active field in natural language processing with many successful developed models in recent years. Despite their high in-distribution accuracy, these models suffer from two issues: high training cost and low out-of-distribution accuracy. Even though some approaches have been presented to tackle the generalization problem, they have high, intolerable training costs. In this paper, we investigate the effect of ensemble learning approach to improve generalization of MRC systems without retraining a big model. After separately training the base models with different structures on different datasets, they are ensembled using weighting and stacking approaches in probabilistic and non-probabilistic settings. Three configurations are investigated including heterogeneous, homogeneous, and hybrid on eight datasets and six state-of-the-art models. We identify the important factors in the effectiveness of ensemble methods. Also, we compare the robustness of ensemble and fine-tuned models against data distribution shifts. The experimental results show the effectiveness and robustness of the ensemble approach in improving the out-of-distribution accuracy of MRC systems, especially when the base models are similar in accuracies.