Evolutionary Feature-wise Thresholding for Binary Representation of NLP Embeddings

TL;DR

This paper introduces a coordinate search-based method for optimizing feature-specific thresholds to convert NLP embeddings into binary representations, improving accuracy and efficiency in large-scale NLP tasks.

Contribution

It proposes a novel thresholding framework that identifies optimal per-feature thresholds, enhancing binary encoding performance over fixed-threshold methods.

Findings

Binary embeddings with optimized thresholds outperform traditional methods in accuracy.

The approach improves efficiency and accuracy in NLP applications.

The method is versatile and applicable beyond NLP embeddings.

Abstract

Efficient text embedding is crucial for large-scale natural language processing (NLP) applications, where storage and computational efficiency are key concerns. In this paper, we explore how using binary representations (barcodes) instead of real-valued features can be used for NLP embeddings derived from machine learning models such as BERT. Thresholding is a common method for converting continuous embeddings into binary representations, often using a fixed threshold across all features. We propose a Coordinate Search-based optimization framework that instead identifies the optimal threshold for each feature, demonstrating that feature-specific thresholds lead to improved performance in binary encoding. This ensures that the binary representations are both accurate and efficient, enhancing performance across various features. Our optimal barcode representations have shown promising results in various NLP applications, demonstrating their potential to transform text representation. We conducted extensive experiments and statistical tests on different NLP tasks and datasets to evaluate our approach and compare it to other thresholding methods. Binary embeddings generated using using optimal thresholds found by our method outperform traditional binarization methods in accuracy. This technique for generating binary representations is versatile and can be applied to any features, not just limited to NLP embeddings, making it useful for a wide range of domains in machine learning applications.