Optimal Size-Performance Tradeoffs: Weighing PoS Tagger Models

TL;DR

This paper introduces methods to measure and compare NLP model size and performance, applying them to POS taggers across eight languages to identify optimal trade-offs and challenging the assumption that bigger models always perform better.

Contribution

It presents novel techniques for quantifying model size and performance trade-offs, and applies them to POS tagging, revealing classical models often remain optimal despite deep models' higher scores.

Findings

Classical POS taggers often lie on the size-performance skyline.

Deep models achieve higher scores but are not always the most size-efficient.

Some models balance size and performance better than larger, more complex models.

Abstract

Improvement in machine learning-based NLP performance are often presented with bigger models and more complex code. This presents a trade-off: better scores come at the cost of larger tools; bigger models tend to require more during training and inference time. We present multiple methods for measuring the size of a model, and for comparing this with the model's performance. In a case study over part-of-speech tagging, we then apply these techniques to taggers for eight languages and present a novel analysis identifying which taggers are size-performance optimal. Results indicate that some classical taggers place on the size-performance skyline across languages. Further, although the deep models have highest performance for multiple scores, it is often not the most complex of these that reach peak performance.