Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraints

TL;DR

This paper introduces DA-GRPO, a novel reinforcement learning method that enables local language models to learn task competence and collaborate with cloud models efficiently under strict budget constraints, improving accuracy and reducing forgetting.

Contribution

The paper presents DA-GRPO, a new policy optimization algorithm that directly incorporates cloud-usage constraints into continual learning of local language models.

Findings

DA-GRPO improves accuracy after task switches.

It significantly reduces catastrophic forgetting.

It maintains stable cloud offloading within budget constraints.

Abstract

Locally deployed Small Language Models (SLMs) must continually support diverse tasks under strict memory and computation constraints, making selective reliance on cloud Large Language Models (LLMs) unavoidable. Regulating cloud assistance during continual learning is challenging, as naive reward-based reinforcement learning often yields unstable offloading behavior and exacerbates catastrophic forgetting as task distributions shift. We propose DA-GRPO, a dual-advantage extension of Group Relative Policy Optimization that incorporates cloud-usage constraints directly into advantage computation, avoiding fixed reward shaping and external routing models. This design enables the local model to jointly learn task competence and collaboration behavior, allowing cloud requests to emerge naturally during post-training while respecting a prescribed assistance budget. Experiments on mathematical reasoning and code generation benchmarks show that DA-GRPO improves post-switch accuracy, substantially reduces forgetting, and maintains stable cloud usage compared to prior collaborative and routing-based approaches.