Context-Aware CodeLLM Eviction for AI-assisted Coding

TL;DR

This paper introduces CACE, a context-aware eviction strategy for self-hosted CodeLLMs that improves latency and resource efficiency by considering multiple factors beyond recency, tailored for AI-assisted coding workflows.

Contribution

The paper proposes CACE, a novel multi-factor eviction algorithm that enhances model management for self-hosted CodeLLMs under resource constraints, outperforming traditional methods.

Findings

CACE reduces latency and model eviction frequency.

Multi-factor eviction balances responsiveness and efficiency.

Experimental results outperform state-of-the-art systems.

Abstract

AI-assisted coding tools powered by Code Large Language Models (CodeLLMs) are increasingly integrated into modern software development workflows. To address concerns around privacy, latency, and model customization, many enterprises opt to self-host these models. However, the diversity and growing number of CodeLLMs, coupled with limited accelerator memory, introduce practical challenges in model management and serving efficiency. This paper presents CACE, a novel context-aware model eviction strategy designed specifically to optimize self-hosted CodeLLM serving under resource constraints. Unlike traditional eviction strategies based solely on recency (e.g., Least Recently Used), CACE leverages multiple context-aware factors, including model load time, task-specific latency sensitivity, expected output length, and recent usage and future demand tracked through a sliding window. We evaluate CACE using realistic workloads that include both latency-sensitive code completion and throughput-intensive code reasoning tasks. Our experiments show that CACE reduces Time-to-First-Token (TTFT) and end-to-end (E2E) latency, while significantly lowering the number of model evictions compared to state-of-the-art systems. Ablation studies further demonstrate the importance of multi-factor eviction in balancing responsiveness and resource efficiency. This work contributes practical strategies for deploying scalable, low-latency AI coding assistants in real-world software engineering environments.