Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens

TL;DR

This paper explores using attention mechanisms in neural networks trained only on sentence labels to infer token-level labels, demonstrating that attention-based methods can effectively identify relevant tokens, sometimes rivaling supervised models.

Contribution

Introduces a neural network architecture with soft attention for zero-shot token labeling, enabling token-level inference from sentence-level training.

Findings

Attention-based methods outperform gradient-based techniques in token prediction.

Attention methods sometimes rival supervised oracle models.

Provides a quantitative evaluation of what models learn at token level.

Abstract

Can attention- or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level labels? We construct a neural network architecture based on soft attention, train it as a binary sentence classifier and evaluate against token-level annotation on four different datasets. Inferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating useful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, sometimes even rivaling the supervised oracle network.