Iterative Definition Refinement for Zero-Shot Classification via LLM-Based Semantic Prototype Optimization

TL;DR

This paper introduces an iterative, LLM-driven method to refine category definitions for zero-shot web content classification, significantly enhancing accuracy without retraining models.

Contribution

It proposes a training-free, adaptive framework that optimizes category definitions iteratively using LLMs, improving zero-shot classification performance.

Findings

Iterative refinement improves classification accuracy across models.

The approach reduces semantic overlap caused by ambiguous definitions.

A new benchmark dataset with 10 URL categories and 1,000 samples per class is introduced.

Abstract

Web filtering systems rely on accurate web content classification to block cyber threats, prevent data exfiltration, and ensure compliance. However, classification is increasingly difficult due to the dynamic and rapidly evolving nature of the modern web. Embedding-based zero-shot approaches map content and category descriptions into a shared semantic space, enabling label assignment without labeled training data, but remain highly sensitive to definition quality. Poorly specified or ambiguous definitions create semantic overlap in the embedding space, leading to systematic misclassification. In this paper, we propose a training-free, adaptive iterative definition refinement framework that improves zero-shot web content classification by progressively optimizing category definitions rather than updating model parameters. Using LLMs as feedback-driven definition optimizers, we investigate three refinement strategies namely example-guided, confusion-aware, and history-aware, each refining class descriptions using structured signals from misclassified instances. Furthermore, we introduce a human-labeled benchmark of 10 URL categories with 1,000 samples per class and evaluate across 13 state-of-the-art embedding foundation models. Results demonstrate that iterative definition refinement consistently improves classification performance across diverse architectures, establishing definition quality as a critical and underexplored factor in embedding-based systems. The dataset is available at https://github.com/naeemrehmat/B2MWT-10C.