# Smaller Text Classifiers with Discriminative Cluster Embeddings

## TL;DR

This paper introduces a method to significantly reduce text classifier sizes by learning discrete word clusters end-to-end, using Gumbel-Softmax, and selectively adding parameters for improved accuracy.

## Contribution

It presents a novel end-to-end clustering approach with Gumbel-Softmax for smaller, more efficient text classifiers, including parameter-efficient variations.

## Key findings

- Reduced model size with maintained accuracy
- Effective end-to-end clustering of words
- Parameter-efficient improvements

## Abstract

Word embedding parameters often dominate overall model sizes in neural methods for natural language processing. We reduce deployed model sizes of text classifiers by learning a hard word clustering in an end-to-end manner. We use the Gumbel-Softmax distribution to maximize over the latent clustering while minimizing the task loss. We propose variations that selectively assign additional parameters to words, which further improves accuracy while still remaining parameter-efficient.