BERMo: What can BERT learn from ELMo?

TL;DR

BERMo enhances BERT by integrating hierarchical surface, syntactic, and semantic features using a simple combination scheme, leading to better accuracy, faster convergence, and improved parameter efficiency with minimal added complexity.

Contribution

The paper introduces BERMo, a novel modification to BERT that incorporates hierarchical features via a linear combination scheme inspired by ELMo, improving gradient flow and representation power.

Findings

Up to 4.65% accuracy improvement on SentEval semantic tasks.

Faster convergence by 1.67x and 1.15x on MNLI and QQP.

Enables stable pruning for small datasets like SST-2.

Abstract

We propose BERMo, an architectural modification to BERT, which makes predictions based on a hierarchy of surface, syntactic and semantic language features. We use linear combination scheme proposed in Embeddings from Language Models (ELMo) to combine the scaled internal representations from different network depths. Our approach has two-fold benefits: (1) improved gradient flow for the downstream task as every layer has a direct connection to the gradients of the loss function and (2) increased representative power as the model no longer needs to copy the features learned in the shallower layer which are necessary for the downstream task. Further, our model has a negligible parameter overhead as there is a single scalar parameter associated with each layer in the network. Experiments on the probing task from SentEval dataset show that our model performs up to $4.65\%$ better in accuracy than the baseline with an average improvement of $2.67\%$ on the semantic tasks. When subject to compression techniques, we find that our model enables stable pruning for compressing small datasets like SST-2, where the BERT model commonly diverges. We observe that our approach converges $1.67\times$ and $1.15\times$ faster than the baseline on MNLI and QQP tasks from GLUE dataset. Moreover, our results show that our approach can obtain better parameter efficiency for penalty based pruning approaches on QQP task.