Evidence > Intuition: Transferability Estimation for Encoder Selection
Elisa Bassignana, Max M\"uller-Eberstein, Mike Zhang, Barbara, Plank

TL;DR
This paper introduces a method to predict the best pre-trained language model for a specific NLP task using transferability estimation, reducing the need for extensive fine-tuning and outperforming intuition-based selection.
Contribution
It applies the Logarithm of Maximum Evidence (LogME) measure from computer vision to NLP, demonstrating its effectiveness in ranking language models across multiple tasks.
Findings
LogME correlates with model performance in 94% of cases
Quantitative transferability measures outperform human intuition in model selection
Evidence-based ranking can identify unexpected suitable models
Abstract
With the increase in availability of large pre-trained language models (LMs) in Natural Language Processing (NLP), it becomes critical to assess their fit for a specific target task a priori - as fine-tuning the entire space of available LMs is computationally prohibitive and unsustainable. However, encoder transferability estimation has received little to no attention in NLP. In this paper, we propose to generate quantitative evidence to predict which LM, out of a pool of models, will perform best on a target task without having to fine-tune all candidates. We provide a comprehensive study on LM ranking for 10 NLP tasks spanning the two fundamental problem types of classification and structured prediction. We adopt the state-of-the-art Logarithm of Maximum Evidence (LogME) measure from Computer Vision (CV) and find that it positively correlates with final LM performance in 94% of the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Data Classification · Explainable Artificial Intelligence (XAI) · Topic Modeling
