Just Pass Twice: Efficient Token Classification with LLMs for Zero-Shot NER

TL;DR

This paper introduces Just Pass Twice (JPT), a simple method enabling causal LLMs to perform zero-shot NER efficiently by using a double-pass input technique for full context access.

Contribution

JPT allows causal LLMs to perform bidirectional token classification without architectural changes, achieving state-of-the-art zero-shot NER results and significantly faster inference.

Findings

Surpassed previous best zero-shot NER by +7.9 F1 on average.

Over 20x faster than comparable generative methods.

Achieved state-of-the-art results on CrossNER and MIT benchmarks.

Abstract

Large language models encode extensive world knowledge valuable for zero-shot named entity recognition. However, their causal attention mechanism, where tokens attend only to preceding context, prevents effective token classification when disambiguation requires future context. Existing approaches use LLMs generatively, prompting them to list entities or produce structured outputs, but suffer from slow autoregressive decoding, hallucinated entities, and formatting errors. We propose Just Pass Twice (JPT), a simple yet effective method that enables causal LLMs to perform discriminative token classification with full bidirectional context. Our key insight is that concatenating the input to itself lets each token in the second pass attend to the complete sentence, requiring no architectural modifications. We combine these representations with definition-guided entity embeddings for flexible zero-shot generalization. Our approach achieves state-of-the-art results on zero-shot NER benchmarks, surpassing the previous best method by +7.9 F1 on average across CrossNER and MIT benchmarks, being over 20x faster than comparable generative methods.