Custom 8-bit floating point value format for reducing shared memory bank conflict in approximate nearest neighbor search

TL;DR

This paper introduces a custom 8-bit floating point format designed to reduce shared memory bank conflicts in approximate nearest neighbor search on GPUs, improving throughput with minimal impact on recall.

Contribution

The paper proposes a novel 8-bit floating point format without a sign bit, optimized for GPU-based ANNS, enhancing performance over standard formats.

Findings

Achieved higher search throughput on GPUs using the custom format.

Maintained comparable recall rates to FP32 and FP16 formats.

Reduced shared memory bank conflicts significantly.

Abstract

The k-nearest neighbor search is used in various applications such as machine learning, computer vision, database search, and information retrieval. While the computational cost of the exact nearest neighbor search is enormous, an approximate nearest neighbor search (ANNS) has been attracting much attention. IVFPQ is one of the ANNS methods. Although we can leverage the high bandwidth and low latency of shared memory to compute the search phase of the IVFPQ on NVIDIA GPUs, the throughput can degrade due to shared memory bank conflict. To reduce the bank conflict and improve the search throughput, we propose a custom 8-bit floating point value format. This format doesn't have a sign bit and can be converted from/to FP32 with a few instructions. We use this format for IVFPQ on GPUs and achieved better performance without significant recall loss compared to FP32 and FP16.