Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updates

TL;DR

This paper introduces LiveUpdate, a system that enables near-zero-overhead, real-time model updates for recommendation systems by colocating low-rank trainers with inference nodes, significantly improving freshness and accuracy.

Contribution

LiveUpdate leverages low-rank structure and resource scheduling to eliminate synchronization overhead, enabling continuous online model updates in recommendation systems.

Findings

Reduces update costs by 2x compared to delta-update baselines.

Achieves higher recommendation accuracy within 1-hour update windows.

Transforms idle inference resources into real-time freshness engines.

Abstract

Deep Learning Recommendation Models (DLRMs) underpin personalized services but face a critical freshness-accuracy tradeoff due to massive parameter synchronization overheads. Production DLRMs deploy decoupled training/inference clusters, where synchronizing petabyte-scale embedding tables (EMTs) causes multi-minute staleness, degrading recommendation quality and revenue. We observe that (1) inference nodes exhibit sustained CPU underutilization (peak <= 20%), and (2) EMT gradients possess intrinsic low-rank structure, enabling compact update representation. We present LiveUpdate, a system that eliminates inter-cluster synchronization by colocating Low-Rank Adaptation (LoRA) trainers within inference nodes. LiveUpdate addresses two core challenges: (1) dynamic rank adaptation via singular value monitoring to constrain memory overhead (<2% of EMTs), and (2) NUMA-aware resource scheduling with hardware-enforced QoS to eliminate update inference contention (P99 latency impact <20ms). Evaluations show LiveUpdate reduces update costs by 2x versus delta-update baselines while achieving higher accuracy within 1-hour windows. By transforming idle inference resources into freshness engines, LiveUpdate delivers online model updates while outperforming state-of-the-art delta-update methods by 0.04% to 0.24% in accuracy.