Systematic Evaluation of Optimization Techniques for Long-Context Language Models

TL;DR

This paper systematically evaluates various optimization techniques for long-context large language models, analyzing their impact on performance, accuracy, and scalability across different architectures and model sizes.

Contribution

It provides a comprehensive benchmarking framework for optimization methods, revealing their effects on large models and offering insights into effective combinations and trade-offs.

Findings

Naive optimization combinations can harm larger models due to approximation errors.

F1 score alone can mask important precision-recall trade-offs.

System-level profiling combined with task insights aids in balancing efficiency and accuracy.

Abstract

Large language models (LLMs) excel across diverse natural language processing tasks but face resource demands and limited context windows. Although techniques like pruning, quantization, and token dropping can mitigate these issues, their efficacy in long-context scenarios and system evaluation remains underexplored. This paper systematically benchmarks these optimizations, characterizing memory usage, latency, and throughput, and studies how these methods impact the quality of text generation. We first analyze individual optimization methods for two LLM architectures supporting long context and then systematically evaluate combinations of these techniques to assess how this deeper analysis impacts performance metrics. We subsequently study the scalability of individual optimization methods on a larger variant with 70 billion-parameter model. Our novel insights reveal that naive combination inference optimization algorithms can adversely affect larger models due to compounded approximation errors, as compared to their smaller counterparts. Experiments show that relying solely on F1 obscures these effects by hiding precision-recall trade-offs in question answering tasks. By integrating system-level profiling with task-specific insights, this study helps LLM practitioners and researchers explore and balance efficiency, accuracy, and scalability across tasks and hardware configurations.