GLiClass: Generalist Lightweight Model for Sequence Classification Tasks

TL;DR

GLiClass introduces a versatile, efficient, and accurate lightweight model for sequence classification that excels in zero-shot and few-shot scenarios, addressing limitations of generative LLMs and cross-encoders.

Contribution

The paper presents GLiClass, a novel adaptation of the GLiNER architecture for sequence classification, combining high efficiency with strong zero-shot and few-shot capabilities.

Findings

Achieves accuracy comparable to embedding-based methods

Maintains efficiency in large label set scenarios

Effective in data-sparse and human feedback settings

Abstract

Classification is one of the most widespread tasks in AI applications, serving often as the first step in filtering, sorting, and categorizing data. Since modern AI systems must handle large volumes of input data and early pipeline stages can propagate errors downstream, achieving high efficiency and accuracy is critical. Moreover, classification requirements can change dynamically based on user needs, necessitating models with strong zero-shot capabilities. While generative LLMs have become mainstream for zero-shot classification due to their versatility, they suffer from inconsistent instruction following and computational inefficiency. Cross-encoders, commonly used as rerankers in RAG pipelines, face a different bottleneck: they must process text-label pairs sequentially, significantly reducing efficiency with large label sets. Embedding-based approaches offer good efficiency but struggle with complex scenarios involving logical and semantic constraints. We propose GLiClass, a novel method that adapts the GLiNER architecture for sequence classification tasks. Our approach achieves strong accuracy and efficiency comparable to embedding-based methods, while maintaining the flexibility needed for zero-shot and few-shot learning scenarios. Additionally, we adapted proximal policy optimization (PPO) for multi-label text classification, enabling training classifiers in data-sparse conditions or from human feedback.