Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers

TL;DR

This paper proposes a soft attention module that enhances zero-shot sequence labeling in transformer-based sentence classifiers by sharpening attention weights, leading to improved token-level performance without supervision.

Contribution

It introduces a novel soft attention mechanism that explicitly encourages sharpness in attention weights, significantly improving zero-shot token labeling in transformers.

Findings

Soft attention improves zero-shot sequence labeling performance.

Explicitly encouraging attention sharpness outperforms existing methods.

Transformer-based models benefit from the proposed attention mechanism.

Abstract

We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-based architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.