The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment

TL;DR

This paper investigates the 'framework tax', a discrepancy where increased computational efficiency in NLP models does not lead to faster inference times due to framework bottlenecks, affecting deployment.

Contribution

The study introduces the concept of the framework tax, analyzes its impact on inference latency, and highlights how model design, frameworks, and hardware influence this disparity.

Findings

Framework bottlenecks significantly increase inference latency.

Disparity between computational throughput and real-world latency is growing.

Analysis of model and hardware factors reveals key sources of the framework tax.

Abstract

Increased focus on the computational efficiency of NLP systems has motivated the design of efficient model architectures and improvements to underlying hardware accelerators. However, the resulting increases in computational throughput and reductions in floating point operations have not directly translated to improvements in wall-clock inference latency. We demonstrate that these discrepancies can be largely attributed to bottlenecks introduced by deep learning frameworks. We denote this phenomenon as the \textit{framework tax}, and observe that the disparity is growing as hardware speed increases over time. In this work, we examine this phenomenon through a series of case studies analyzing the effects of model design decisions, framework paradigms, and hardware platforms on total model latency. Code is available at https://github.com/JaredFern/Framework-Tax.