BERTVision -- A Parameter-Efficient Approach for Question Answering

TL;DR

BERTVision introduces a parameter-efficient method for question answering that leverages hidden layer activations of BERT, reducing training costs and data requirements while maintaining high performance and extending to other tasks.

Contribution

The paper proposes a novel approach that utilizes BERT's hidden states for QA, significantly reducing fine-tuning parameters and training time, and demonstrating versatility across tasks.

Findings

Achieves high BERT performance with less training data.

Ensembling improves QA accuracy.

Effective for span annotation and classification.

Abstract

We present a highly parameter efficient approach for Question Answering that significantly reduces the need for extended BERT fine-tuning. Our method uses information from the hidden state activations of each BERT transformer layer, which is discarded during typical BERT inference. Our best model achieves maximal BERT performance at a fraction of the training time and GPU or TPU expense. Performance is further improved by ensembling our model with BERTs predictions. Furthermore, we find that near optimal performance can be achieved for QA span annotation using less training data. Our experiments show that this approach works well not only for span annotation, but also for classification, suggesting that it may be extensible to a wider range of tasks.