Bad Form: Comparing Context-Based and Form-Based Few-Shot Learning in Distributional Semantic Models

TL;DR

This paper compares context-based and form-based few-shot learning methods in distributional semantic models, introduces new evaluation tasks, and demonstrates that hyperparameter tuning significantly improves model performance, setting new benchmarks.

Contribution

It introduces three new tasks for better evaluation of form-based models and highlights the importance of hyperparameter tuning in improving model performance.

Findings

Form-based models can leverage word form information in training data.

Hyperparameter tuning improves performance across models.

Achieved state-of-the-art results on 4 out of 6 tasks.

Abstract

Word embeddings are an essential component in a wide range of natural language processing applications. However, distributional semantic models are known to struggle when only a small number of context sentences are available. Several methods have been proposed to obtain higher-quality vectors for these words, leveraging both this context information and sometimes the word forms themselves through a hybrid approach. We show that the current tasks do not suffice to evaluate models that use word-form information, as such models can easily leverage word forms in the training data that are related to word forms in the test data. We introduce 3 new tasks, allowing for a more balanced comparison between models. Furthermore, we show that hyperparameters that have largely been ignored in previous work can consistently improve the performance of both baseline and advanced models, achieving a new state of the art on 4 out of 6 tasks.