AgentCompress: Task-Aware Compression for Affordable Large Language Model Agents

TL;DR

AgentCompress is a task-aware dynamic compression framework that routes language model queries to different model sizes based on task complexity, significantly reducing costs while maintaining high accuracy across diverse domains.

Contribution

It introduces a lightweight neural controller for task-aware model routing, enabling cost-effective use of large language models without sacrificing performance.

Findings

68.3% reduction in computational costs

96.2% of original success rate preserved

Effective across multiple scientific domains

Abstract

Large language models hold considerable promise for various applications, but their computational requirements create a barrier that many institutions cannot overcome. A single session using a 70-billion-parameter model can cost around $127 in cloud computing fees, which puts these tools out of reach for organizations operating on limited budgets. We present AgentCompress, a framework that tackles this problem through task-aware dynamic compression. The idea comes from a simple observation: not all tasks require the same computational effort. Complex reasoning, for example, is far more demanding than text reformatting, yet conventional compression applies the same reduction to both. Our approach uses a lightweight neural controller that looks at the first few tokens of each request, estimates how complex the task will be, and sends it to an appropriately quantized version of the model. This routing step adds only about 12 milliseconds of overhead. We tested the framework on 290 multi-stage workflows from domains including computer science, physics, chemistry, and biology. The results show a 68.3% reduction in computational costs while preserving 96.2% of the original success rate. These findings suggest that routing queries intelligently can make powerful language models substantially more affordable without sacrificing output quality