LLM-based Affective Text Generation Quality Based on Different Quantization Values

TL;DR

This study investigates how different quantization levels affect the quality, memory usage, and inference time of large language models in affective text generation, highlighting trade-offs between efficiency and accuracy.

Contribution

It provides a comprehensive analysis of quantization impacts on affective text generation across multiple models, revealing memory savings and accuracy trade-offs.

Findings

76% memory reduction with quantization

Up to 10 percentage points decrease in F1 score for larger models

Inference time roughly doubles with lower precision

Abstract

Large language models exhibit a remarkable capacity in language generation and comprehension. These advances enable AI systems to produce more human-like and emotionally engaging text. However, these models rely on a large number of parameters, requiring significant computational resources for training and inference. In some scenarios, accessing these resources can be challenging (e.g., budget or hardware limitations). Techniques like reducing precision bits can make models more memory-efficient, reducing the computational resources needed, at the cost of reduced accuracy. This paper addresses the trade-off between different quantization values, GPU RAM utilization, and text quality in affective text generation (e.g., "I really enjoy running in the snow-covered forest"). To evaluate, we use an emotion classifier and ten seed prompts to generate affective text. We test three setups of precision bits (8, 16, and 32) across five open-weight language models from two different families. Our findings demonstrate that bit reductions lead to memory savings, achieving a reduction of 76%. However, this optimization comes with a trade-off, leading to a decrease of up to 10 pp in F1 score for larger models and an increase of 10 pp for smaller models, along with roughly double the inference time. In terms of text quality, larger models at lower quantization levels generally outperform smaller, higher-precision models -- while requiring similar memory.