LOCUS: A System and Method for Low-Cost Customization for Universal Specialization

TL;DR

LOCUS is a low-cost, efficient NLP pipeline that leverages few-shot data, retrieval, synthetic data, and parameter-efficient tuning to outperform larger models on NER and text classification tasks.

Contribution

Introduces LOCUS, a novel pipeline combining retrieval, synthetic data, and low-rank tuning for cost-effective NLP model customization with minimal data.

Findings

Outperforms strong baselines including GPT-4o on NER and TC benchmarks.

Achieves 99% of fully fine-tuned accuracy with only 5% of memory.

Uses less than 1% of GPT-4o's parameters while outperforming it.

Abstract

We present LOCUS (LOw-cost Customization for Universal Specialization), a pipeline that consumes few-shot data to streamline the construction and training of NLP models through targeted retrieval, synthetic data generation, and parameter-efficient tuning. With only a small number of labeled examples, LOCUS discovers pertinent data in a broad repository, synthesizes additional training samples via in-context data generation, and fine-tunes models using either full or low-rank (LoRA) parameter adaptation. Our approach targets named entity recognition (NER) and text classification (TC) benchmarks, consistently outperforming strong baselines (including GPT-4o) while substantially lowering costs and model sizes. Our resultant memory-optimized models retain 99% of fully fine-tuned accuracy while using barely 5% of the memory footprint, also beating GPT-4o on several benchmarks with less than 1% of its parameters.