Simple is Better and Large is Not Enough: Towards Ensembling of Foundational Language Models

TL;DR

This paper demonstrates that ensembling smaller foundational language models like BERT with novel techniques can outperform larger models, reducing hallucination and uncertainty in NLP tasks, especially in sensitive applications.

Contribution

It introduces a new ensemble approach, including a knowledge-guided deep ensemble, showing smaller models can surpass larger ones in performance.

Findings

Deep-Ensemble BERT outperforms BERTlarge on benchmark datasets.

Ensembling enhances the coordination among FLMs, improving NLP task accuracy.

Smaller models with ensembling reduce hallucination and uncertainty.

Abstract

Foundational Language Models (FLMs) have advanced natural language processing (NLP) research. Current researchers are developing larger FLMs (e.g., XLNet, T5) to enable contextualized language representation, classification, and generation. While developing larger FLMs has been of significant advantage, it is also a liability concerning hallucination and predictive uncertainty. Fundamentally, larger FLMs are built on the same foundations as smaller FLMs (e.g., BERT); hence, one must recognize the potential of smaller FLMs which can be realized through an ensemble. In the current research, we perform a reality check on FLMs and their ensemble on benchmark and real-world datasets. We hypothesize that the ensembling of FLMs can influence the individualistic attention of FLMs and unravel the strength of coordination and cooperation of different FLMs. We utilize BERT and define three other ensemble techniques: {Shallow, Semi, and Deep}, wherein the Deep-Ensemble introduces a knowledge-guided reinforcement learning approach. We discovered that the suggested Deep-Ensemble BERT outperforms its large variation i.e. BERTlarge, by a factor of many times using datasets that show the usefulness of NLP in sensitive fields, such as mental health.