Opt-GPTQ: An Optimized GPTQ Combining Sparse Attention and Quantization Techniques

TL;DR

Opt-GPTQ introduces an optimized attention mechanism combining grouping, sharing, and quantization techniques to significantly improve efficiency and scalability of large-scale models in deep learning.

Contribution

It proposes a novel combination of grouped query attention and quantization, optimizing attention mechanisms for better performance and resource utilization in large models.

Findings

Reduces computation time and memory usage

Enhances long-sequence processing capabilities

Improves model performance with optimized attention

Abstract

In the field of deep learning, traditional attention mechanisms face significant challenges related to high computational complexity and large memory consumption when processing long sequence data. To address these limitations, we propose Opt-GPTQ, an optimized Gradient-based Post Training Quantization (GPTQ) combining the Grouped Query Attention (GQA) mechanism with paging memory management, optimizing the traditional Multi-Head Attention (MHA) mechanism by grouping query heads and sharing key-value vectors. Optimized GQA (Opt-GQA) effectively reduces computational complexity, minimizes memory fragmentation, and enhances memory utilization for large-scale models. Opt-GPTQ is optimized for Data Center Units (DCUs) and integrated into the vLLM model to maximize hardware efficiency. It customizes GPU kernels to further enhance attention computation by reducing memory access latency and boosting parallel computing capabilities. Opt-GQA integrates Attention with Linear Biases (ALiBi) to reduce overhead and enhance long-sequence processing. Experimental results show that Opt-GPTQ significantly reduces computation time and memory usage while improving model performance.