Exploring BERT Parameter Efficiency on the Stanford Question Answering Dataset v2.0

TL;DR

This paper investigates the parameter efficiency of BERT on SQuAD2.0 by freezing layers, adding adapter and convolutional layers, and evaluating resource use including training and inference times.

Contribution

It systematically evaluates BERT's parameter efficiency with various modifications and introduces context-aware convolutional filters, providing insights into resource trade-offs.

Findings

Adapter layers improve parameter efficiency consistent with prior work.

Adding convolutional filters does not yield practical performance gains.

Efficiency metrics like training and inference time are crucial for model evaluation.

Abstract

In this paper we explore the parameter efficiency of BERT arXiv:1810.04805 on version 2.0 of the Stanford Question Answering dataset (SQuAD2.0). We evaluate the parameter efficiency of BERT while freezing a varying number of final transformer layers as well as including the adapter layers proposed in arXiv:1902.00751. Additionally, we experiment with the use of context-aware convolutional (CACNN) filters, as described in arXiv:1709.08294v3, as a final augmentation layer for the SQuAD2.0 tasks. This exploration is motivated in part by arXiv:1907.10597, which made a compelling case for broadening the evaluation criteria of artificial intelligence models to include various measures of resource efficiency. While we do not evaluate these models based on their floating point operation efficiency as proposed in arXiv:1907.10597, we examine efficiency with respect to training time, inference time, and total number of model parameters. Our results largely corroborate those of arXiv:1902.00751 for adapter modules, while also demonstrating that gains in F1 score from adding context-aware convolutional filters are not practical due to the increase in training and inference time.