Cost-effective Deployment of BERT Models in Serverless Environment

TL;DR

This paper demonstrates that BERT models can be effectively deployed in serverless environments using knowledge distillation and fine-tuning, achieving acceptable latency and cost-effectiveness for real-world NLP tasks.

Contribution

It introduces a method for deploying domain-tuned BERT models in serverless setups without infrastructure overhead, using knowledge distillation and fine-tuning.

Findings

Achieves acceptable latency for production use

Cost-effective for small-to-medium deployments

No additional infrastructure required

Abstract

In this study we demonstrate the viability of deploying BERT-style models to serverless environments in a production setting. Since the freely available pre-trained models are too large to be deployed in this way, we utilize knowledge distillation and fine-tune the models on proprietary datasets for two real-world tasks: sentiment analysis and semantic textual similarity. As a result, we obtain models that are tuned for a specific domain and deployable in serverless environments. The subsequent performance analysis shows that this solution results in latency levels acceptable for production use and that it is also a cost-effective approach for small-to-medium size deployments of BERT models, all without any infrastructure overhead.