When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

TL;DR

This paper demonstrates that 8-bit quantization enhances continual learning in large language models by improving knowledge retention and task performance, challenging the notion that higher precision always yields better results.

Contribution

The study reveals that INT8 quantization, combined with minimal replay buffers, outperforms higher precision models in continual learning, providing new insights into model regularization and deployment efficiency.

Findings

INT8 models outperform FP16 on final task accuracy

Minimal replay buffers significantly improve retention across precisions

Quantization acts as implicit regularization, aiding continual learning

Abstract

Catastrophic forgetting poses a fundamental challenge in continual learning, particularly when models are quantized for deployment efficiency. We systematically investigate the interplay between quantization precision (FP16, INT8, INT4) and replay buffer strategies in large language models, revealing unexpected dynamics. While FP16 achieves superior initial task performance (74.44% on NLU), we observe a striking inversion on subsequent tasks: quantized models outperform FP16 by 8-15% on final task forward accuracy, with INT4 achieving nearly double FP16's performance on Code generation (40% vs 20%). Critically, even minimal replay buffers (0.1%) dramatically improve retention - increasing NLU retention after Math training from 45% to 65% across all precision levels - with INT8 consistently achieving the optimal balance between learning plasticity and knowledge retention. We hypothesize that quantization-induced noise acts as implicit regularization, preventing the overfitting to new task gradients that plagues high-precision models. These findings challenge the conventional wisdom that higher precision is always preferable, suggesting instead that INT8 quantization offers both computational efficiency and superior continual learning dynamics. Our results provide practical guidelines for deploying compressed models in continual learning scenarios: small replay buffers (1-2%) suffice for NLU tasks, while Math and Code benefit from moderate buffers (5-10%), with quantized models requiring less replay than FP16 to achieve comparable retention. Code is available at https://github.com/Festyve/LessIsMore.