cMALC-D: Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending

TL;DR

This paper introduces cMALC-D, a novel curriculum learning framework for multi-agent reinforcement learning that leverages large language models and diversity-based context blending to enhance generalization and efficiency.

Contribution

The paper proposes a new LLM-guided curriculum learning method with a diversity-based context blending mechanism for multi-agent RL, addressing instability and exploration issues.

Findings

cMALC-D outperforms existing curriculum methods in traffic signal control tasks.

The diversity-based blending prevents mode collapse and promotes exploration.

The framework improves both generalization and sample efficiency.

Abstract

Many multi-agent reinforcement learning (MARL) algorithms are trained in fixed simulation environments, making them brittle when deployed in real-world scenarios with more complex and uncertain conditions. Contextual MARL (cMARL) addresses this by parameterizing environments with context variables and training a context-agnostic policy that performs well across all environment configurations. Existing cMARL methods attempt to use curriculum learning to help train and evaluate context-agnostic policies, but they often rely on unreliable proxy signals, such as value estimates or generalized advantage estimates that are noisy and unstable in multi-agent settings due to inter-agent dynamics and partial observability. To address these issues, we propose Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending (cMALC-D), a framework that uses Large Language Models (LLMs) to generate semantically meaningful curricula and provide a more robust evaluation signal. To prevent mode collapse and encourage exploration, we introduce a novel diversity-based context blending mechanism that creates new training scenarios by combining features from prior contexts. Experiments in traffic signal control domains demonstrate that cMALC-D significantly improves both generalization and sample efficiency compared to existing curriculum learning baselines. We provide code at https://github.com/DaRL-LibSignal/cMALC-D.

Tables

Table 1. Table 1 : Performance metrics across all environments. Best results per metric are shown in bold and second-best results are underlined . We include uncertainty within one standard deviation of the mean, averaged over 5 seeds.

Curriculum	Average Time	Throughput	Average Wait Time	Average Delay	Test Reward
No Curriculum	816.00 $\pm$ 35.63	3032.72 $\pm$ 114.20	765.84 $\pm$ 36.76	728.34 $\pm$ 37.56	$27.56\pm 0.46$
Domain Randomization	$865.64\pm 43.08$	$2807.22\pm 141.63$	$816.64\pm 44.41$	$774.43\pm 46.66$	27.55 $\pm$ 0.41
PLR	$841.74\pm 40.63$	$2955.79\pm 127.53$	$795.98\pm 41.65$	$772.72\pm 41.46$	$\underset{¯}{27.61}\pm \underset{¯}{0.44}$
ACCEL	$860.81\pm 39.57$	$2813.86\pm 130.81$	$819.49\pm 40.75$	$806.07\pm 41.06$	$26.55\pm 0.45$
SPACE	$939.39\pm 39.96$	$2544.24\pm 133.01$	$898.71\pm 41.15$	$875.71\pm 41.19$	$26.89\pm 0.43$
cMALC-D	809.39 $\pm$ 36.37	3073.22 $\pm$ 114.06	750.81 $\pm$ 36.99	718.77 $\pm$ 37.26	29.01 $\pm$ 0.32

Table 2. (a) JN 1 × 3 1\times 3 Performance Metrics

Curriculum	Average Time	Throughput	Average Wait Time	Average Delay	Test Reward
No Curriculum	816.00 $\pm$ 35.63	3032.72 $\pm$ 114.20	765.84 $\pm$ 36.76	728.34 $\pm$ 37.56	$27.56\pm 0.46$
Domain Randomization	$865.64\pm 43.08$	$2807.22\pm 141.63$	$816.64\pm 44.41$	$774.43\pm 46.66$	27.55 $\pm$ 0.41
PLR	$841.74\pm 40.63$	$2955.79\pm 127.53$	$795.98\pm 41.65$	$772.72\pm 41.46$	$\underset{¯}{27.61}\pm \underset{¯}{0.44}$
ACCEL	$860.81\pm 39.57$	$2813.86\pm 130.81$	$819.49\pm 40.75$	$806.07\pm 41.06$	$26.55\pm 0.45$
SPACE	$939.39\pm 39.96$	$2544.24\pm 133.01$	$898.71\pm 41.15$	$875.71\pm 41.19$	$26.89\pm 0.43$
cMALC-D	809.39 $\pm$ 36.37	3073.22 $\pm$ 114.06	750.81 $\pm$ 36.99	718.77 $\pm$ 37.26	29.01 $\pm$ 0.32

Table 3. (b) HZ Performance Metrics

Curriculum	Average Time	Throughput	Average Wait Time	Average Delay	Test Reward
No Curriculum	$710.01\pm 42.78$	$2188.94\pm 74.78$	$342.84\pm 51.55$	$339.09\pm 53.28$	$164.04\pm 2.68$
Domain Randomization	$637.52\pm 31.61$	$2318.41\pm 53.49$	$239.36\pm 32.60$	$241.79\pm 35.60$	$168.53\pm 1.44$
PLR	$611.26\pm 26.48$	$2390.56\pm 38.11$	$199.22\pm 23.74$	$175.96\pm 22.20$	$171.85\pm 1.02$
ACCEL	615.02 $\pm$ 25.46	$2393.30\pm 35.37$	$212.28\pm 21.12$	$178.54\pm 19.56$	$171.07\pm 1.17$
SPACE	$\underset{¯}{588.84}\pm \underset{¯}{25.11}$	2440.18 $\pm$ 32.02	166.75 $\pm$ 18.13	140.25 $\pm$ 16.30	172.90 $\pm$ 1.05
cMALC-D	586.86 $\pm$ 25.50	2440.09 $\pm$ 35.45	163.92 $\pm$ 19.71	146.96 $\pm$ 19.93	172.87 $\pm$ 1.03

Table 4. (c) JN 3 × 4 3\times 4 Performance Metrics

Curriculum	Average Time	Throughput	Average Wait Time	Average Delay	Test Reward
No Curriculum	829.92 $\pm$ 36.98	3704.17 $\pm$ 147.39	566.12 $\pm$ 43.18	550.47 $\pm$ 46.03	115.77 $\pm$ 2.44
Domain Randomization	$976.73\pm 37.67$	$3108.67\pm 151.36$	$753.96\pm 45.58$	$756.78\pm 49.21$	$112.38\pm 1.52$
PLR	$992.79\pm 47.30$	$3071.38\pm 188.12$	$781.35\pm 57.70$	$786.23\pm 61.33$	$111.98\pm 1.74$
ACCEL	$1077.27\pm 42.13$	$2699.78\pm 166.09$	$903.94\pm 50.26$	$921.75\pm 52.73$	$110.28\pm 1.60$
SPACE	$899.94\pm 43.34$	$3447.63\pm 172.27$	$658.07\pm 51.30$	$638.21\pm 55.47$	$114.47\pm 1.64$
cMALC-D	815.81 $\pm$ 39.51	3795.46 $\pm$ 159.50	557.96 $\pm$ 46.57	529.83 $\pm$ 50.28	116.57 $\pm$ 1.53

Table 5. Table 2 : Data statistics of real-world traffic datasets. Arrival rates are reported vehicles per 300 seconds.

Dataset	# Intersections	Mean	Std	Max	Min	# Lanes	Time Steps
Hangzhou	16	526.63	86.70	676	256	3	3600
Jinan $\left(3\times 4\right)$	12	250.70	38.21	335	208	3	3600
Jinan $\left(1\times 3\right)$	3	278.23	18.47	322	229	3	3600

Table 6. Table 3 : Context parameters used for curriculum learning and their specified ranges.

Parameter	Description	Range
length	Length of each car	$1$–$10$ m
width	Width of each car	$1$–$5$ m
maxPosAcc	Max acceleration when speeding up	$0.5$–$5$ m/s2
maxNegAcc	Max deceleration when braking	$0.5$–$5$ m/s2
usualPosAcc	Default acceleration when speeding up	$1$–$5$ m/s2
usualNegAcc	Default deceleration when braking	$1$–$5$ m/s2
minGap	Minimum gap between cars	$1$–$10$ m
maxSpeed	Maximum speed a car can travel	$3$–$15$ m/s
headwayTime	Desired time to reach the vehicle in front	$1$–$5$ s (int)

Table 7. Table 4 : Hyperparameter for MAPPO

MAPPO Hyperparameters
Eps Clip	0.2
Epsilon Anneal Time	180000
Learning Rate (lr)	0.0003
Hidden Dim	128
Mini Epochs	4
Entropy Coef	0.001
Target Update Interval	0.01
Batch Size	1
Buffer Size	10

Table 8. Table 5 : cMALC-D Hyperparameters

Hyperparameter
$A$	MAPPO
$M$	Qwen2.5-7B-Instruct-AWQ
$\alpha$	0.5
$w$	3
$\delta$	0.1
$k$	3

Table 9. Table 6 : LLM Hyperparameters

Hyperparameter
Temperature	0.7
Top-p	0.9
Max New Tokens	400

Table 10. Table 7 : The following table contains the prompts used for the LLM, both initially and after receiving data from the environments.

Equations

$$\displaystyle\pi^{*}=\operatorname*{arg\,max}_{\pi\in\Pi}\mathbb{E}_{c\sim\mathcal{C}}\left[\mathbb{E}_{\pi}\left[\sum_{t=0}^{H-1}\gamma^{t}\mathcal{R}(s_{t},a_{t}){\bigg{|}}\pi,c\right]\right]$$

Full text

Machine Learning, ICML

1 Introduction

Multi-Agent Reinforcement Learning (MARL) has shown promising results across diverse applications, including real-time strategy games (Samvelyan et al., 2019; Kurach et al., 2020), supply chain management (Liu et al., 2022; Mousa et al., 2024), navigation and pathfinding (Zhang et al., 2024; Skrynnik et al., 2024), and traffic signal control (Chu et al., 2019; Jiang et al., 2022; Satheesh & Powell, 2025). These successes are largely attributed to the ability of MARL algorithms, such as Independent Proximal Policy Optimization (IPPO) (De Witt et al., 2020) and Multi-Agent Proximal Policy Optimization (MAPPO) (Yu et al., 2022), to train agents capable of coordination and cooperation.

Despite this progress, generalization remains a key challenge. Most MARL algorithms are trained in simulation environments with fixed or limited variability, making them brittle when deployed in real-world scenarios where conditions are more complex and uncertain. External factors such as noise (Bukharin et al., 2023; He et al., 2023) and dynamic changes (Zhang et al., 2020) can degrade MARL performance substantially. These issues are amplified in multi-agent settings due to the combinatorial explosion of agent interactions, which can destabilize learned policies and exacerbate overfitting to training conditions. Prior work has addressed these challenges through meta-learning (Finn et al., 2017; Mao et al., 2023; Harris et al., 2022; Zhang et al., 2021), invariant representation learning (Chen & Zhang, 2024; McClellan et al., 2024, 2025), and domain adaptation (Tzeng et al., 2020; Da et al., 2024). While effective, these approaches often rely on predefined environment distributions and cannot guarantee robustness to unseen or out-of-distribution scenarios.

To address the challenge of poor generalization to unseen or out-of-distribution environments, we build on the contextual MARL (cMARL) framework (Jayawardana et al., 2024), which explicitly represents environment variability through a context variable $c$ (Hallak et al., 2015). Generalization in cMARL is commonly improved via curriculum learning (Bengio et al., 2009), where agents are trained on contexts that gradually increase in difficulty or novelty (Tobin et al., 2017; Klink et al., 2020; Jiang et al., 2021; Eimer et al., 2021; Parker-Holder et al., 2023). This allows agents to acquire transferable skills incrementally and improves robustness at test time.

While curriculum learning improves generalization in contextual multi-agent reinforcement learning (MARL) by ordering training environment contexts by difficulty or novelty, existing approaches often depend on handcrafted heuristics or static curriculum schedules. These strategies may struggle to adapt to the evolving agents or the complex dependencies among context variables in dynamic environments. To address these limitations, we explore the use of Large Language Models (LLMs) as high-level curriculum designers. Recent advancements in LLMs, such as GPT-4o, Qwen, and Gemini, have shown strong capabilities in reasoning, planning, and abstraction (Song et al., 2023; Yao et al., 2022; Xu et al., 2023; Ma et al., 2023; Dong et al., 2024). These models can operate in diverse domains through in-context learning, suggesting their potential for adaptively generating and sequencing environment contexts based on the agent’s current state and performance.

In this work, we propose Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending (cMALC-D), a novel framework that integrates LLMs into contextual MARL to dynamically generate training curricula. Specifically, in cMALC-D, the LLM acts as a high-level controller that observes the agent’s learning progress and adaptively proposes new environment contexts by reasoning over the space of context variables. To enhance coverage and prevent overfitting to narrow task distributions, we introduce a diversity-based context blending mechanism that mixes previously sampled contexts to construct novel yet meaningful training conditions. This LLM-guided process allows the curriculum to evolve in tandem with agent capabilities, providing more targeted and generalizable training experiences. We evaluate cMALC-D in multiple traffic control scenarios, where environments are naturally high-dimensional and dynamic. Results show that our approach significantly improves generalization to unseen environment contexts with higher sample efficiency compared to existing self-paced or handcrafted curriculum strategies.

Our contributions are outlined as follows:

• •

We introduce Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending (cMALC-D), a framework that leverages Large Language Models (LLMs) to generate semantically meaningful context-based curricula for training MARL agents, improving generalization to unseen environment configurations.

• •

Experiments in multiple traffic-based environments demonstrate that our approach achieves better generalization compared to other self-paced curricula, with higher sample efficiency.

2 Related Work

2.1 Generalization in MARL

Meta Learning One approach to improving MARL generalizability has been to apply meta-learning to MARL. Meta learning develops a good policy initialization that can rapidly adapt to new tasks or environmental configurations with minimal additional training. This can be done with Model-Agnostic Meta-Learning (MAML) algorithms (Finn et al., 2017; Mao et al., 2023) or through context-based latent variables (Wen et al., 2024; Zhang et al., 2021), both of which train over a task distribution to develop transferable inductive biases. However, such methods face significant scalability issues. MAML requires expensive Hessian computations on the order of the number of model parameters or first-order approximations (Fallah et al., 2021; Nichol et al., 2018). Additionally, prior work focuses on multi-agent solution concepts for two-player games in tabular settings (Mao et al., 2023; Harris et al., 2022; Zhang et al., 2022), which are not extendable to many agents parameterized by neural networks.

Equivariance Multi-agent environments exhibit symmetries that can be leveraged to improve generalization using equivariant networks (McClellan et al., 2024, 2025; Chen & Zhang, 2024). To fully exploit such invariances, each agent must be parameterized with architectures that respect these symmetries, such as EGNNs (Satorras et al., 2021), E3-MPNNs (Brandstetter et al., 2021), or E2GN2s (McClellan et al., 2024). However, these methods typically require access to the full global state, limiting their applicability in partially observable settings, which encompass most multi-agent environments.

Sim-to-Real It is often very challenging to train MARL agents on real-world environment data due to limited accessibility, high costs, and the intricate complexities of real-world systems. A common workaround is to train agents in simulated environments instead. However, simulations often fail to capture the full range of noise and unpredictability found in the real world, making it difficult for agents to transfer their skills effectively (Dulac-Arnold et al., 2021). Several techniques have been proposed to address this sim-to-real gap, one notable method being Domain Randomization. Domain Randomization exposes agents to a wide range of settings by randomizing key environment parameters (Zhao et al., 2020). While effective in some cases, it can produce unrealistic environments that fail to teach agents transferable skills, an issue that becomes more pronounced in MARL settings. Additional methods try to minimize the sim-to-real gap by aligning simulation trajectories to real ones (Tzeng et al., 2020), or by developing an inverse model to correct for bias in the simulation environment (Da et al., 2024), but these methods require information from specific real-world environments, which may not be available. Recently, large language model (LLM)-based methods have shown promise in generating more coherent and purposeful environments, but their application has been largely limited to single-agent contexts (Ma et al., 2024; Zala et al., 2024).

2.2 Self-Paced Curriculum Learning

In contrast to prior approaches such as meta-learning, equivariant architectures, and sim-to-real, curriculum learning (Bengio et al., 2009) aims to improve generalization by structuring agents’ learning process, progressing from easier to more challenging tasks. Additional methods explore automated curriculum generation to avoid manual design. For example, Sukhbaatar et al. (2017) leverages self-play to minimize the number of training episodes by generating progressively harder tasks through agent interactions. Dendorfer et al. (2020) and Florensa et al. (2018) use Generative Adversarial Networks (GANs) to create challenging goals tailored to the agent’s capabilities. Portelas et al. (2020) uses a Gaussian Mixture Model (GMM) to model the task space and align a student’s learning trajectory with a teacher-generated curriculum. However, each of these requires an auxiliary model to determine the learnability of a task. Instead, Klink et al. (2020); Eimer et al. (2021) use self-paced learning, which uses agents’ performance to order the curriculum. Thus, each task is tailored to each agent’s abilities and ensures that the learning progress is more self-contained.

3 Contextual Multi-Agent Reinforcement Learning

We formulate cMALC-D as a contextual decentralized partially observable Markov decision process (cDec-POMDP). A cDec-POMDP is parameterized by the tuple $M_{c}=(\mathcal{N},\mathcal{S},\mathcal{A},\mathcal{T}_{c},R,\Omega,\mathcal{O},\gamma,\mu)$, where $\mathcal{N}$ is the set of agent indices, denoting a system of $n=|\mathcal{N}|$ cooperative agents. $\mathcal{S}$ is the joint state space shared across all agents, and $\mathcal{A}=\prod_{i\in\mathcal{N}}\mathcal{A}^{i}$ is the joint action space, where $\mathcal{A}^{i}$ is the action space for agent $i$. The transition function $\mathcal{T}_{c}:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ determines the next state distribution given the current state and joint action under context $c$, while the reward function $R:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ maps state-action pairs to a scalar reward. $\Omega=\prod_{i\in\mathcal{N}}\Omega^{i}$ denotes the joint observation function, where $\Omega^{i}:\mathcal{S}\rightarrow\mathcal{O}^{i}$ provides a private observation to agent $i$, and $\mathcal{O}=\prod_{i\in\mathcal{N}}\mathcal{O}^{i}$ is the joint observation space. The discount factor $\gamma\in[0,1)$ specifies the importance of future rewards, and $\mu:\mathcal{S}\rightarrow[0,1]$ is the initial state distribution.

To model task variation, we define a distribution over contexts $c\in\mathcal{C}$, where each context specifies a different task instance by altering the transition functions. This induces a set $\mathcal{M}_{\mathcal{C}}=\{M_{c}|c\in\mathcal{C}\}$ of decentralized POMDPs, each corresponding to a distinct environment. Each $M_{c}$ encodes a different task instantiation with a different transition function, and we assume that the reward function and the state, action, and observation spaces remain fixed across all contexts.

The objective of a policy $\pi$ in a cDec-POMDP is to maximize the expected return over the context distribution and over finite horizon $H$:

[TABLE]

where $\Pi=\{\pi=(\pi^{1},\ldots,\pi^{n})\mid\pi^{i}:o^{i}_{t}\rightarrow a^{i}_{t}\}$ is the set of decentralized policies, with each agent $i\in\mathcal{N}$ selecting actions based only on its local observation $o^{i}_{t}$. The goal is to learn a context-agnostic policy that generalizes well across the set of Dec-POMDPs $\mathcal{M}_{\mathcal{C}}$.

4 Evolutionary LLM Self-Paced Curriculum

In this section, we outline the motivations of CMALC-D and its implementation.

4.1 Limitations of Existing Self-Paced Curriculum Learning Algorithms

There are two major limitations of current self-paced curriculum learning algorithms for contextual MARL (cMARL), both of which hinder efficient and robust generalization:

• •

Random Task Sampling Most existing approaches generate new contexts by randomly sampling from the context space, without considering semantic relationships or difficulty progression between sampled environments. This can lead to large variations between contexts across training episodes, making learning unstable and inefficient. For example, an agent might face an easy environment followed by a drastically more difficult or qualitatively different one, forcing it to relearn strategies rather than incrementally building off prior knowledge.

• •

Unreliable Proxy Evaluation Current methods typically rely on policy metrics like the value estimate or the Generalized Advantage Estimate (GAE) to evaluate agent performance and determine subsequent contexts to train on. While these methods work well for single-agent tasks with many environment updates (Jiang et al., 2021; Parker-Holder et al., 2023), these metrics can be unreliable during early training phases and under sudden domain transfer. They can also be overconfident in underexplored regions of the context space and may not provide a meaningful indication of how well agents will generalize to unseen contexts.

These challenges underscore the need for more structured, semantically aware curriculum generation strategies coupled with more robust evaluation signals that better reflect generalization and learning progress across context variations.

4.2 Evolutionary LLM-Guided Curriculum for Context Generation

To address these limitations, we propose cMALC-D, a novel curriculum learning strategy for contextual MARL that combines the structured reasoning capabilities of large language models (LLMs) with an exploration mechanism based on task arithmetic. This approach improves both the generation of semantically meaningful environment contexts and the robustness of policy evaluation under limited feedback.

LLM-Guided Context Generation Instead of randomly sampling from a context space $C$, we leverage a large language model to reason over a sliding window of past training results and generate new contexts that reflect a meaningful progression in difficulty or diversity. At each curriculum step, the LLM receives a window of the most recent contexts $\{c_{t-w},\cdots,c_{t}\}$ and their associated performance metrics $\{m_{t-w},\cdots m_{t}\}$ when trained on MARL algorithm $A$. It then leverages this history to propose a new context that either incrementally challenges the current multi-agent policy or targets known weaknesses observed in recent episodes.

Diversity-Based Context Blending To avoid curriculum stagnation and encourage exploration of the context space, we monitor the similarity between successive contexts. If the LLM repeatedly generates highly similar contexts, indicating potential mode collapse in curriculum progression, we enable a diversity mechanism. Specifically, when the number of consecutive similar tasks exceeds a threshold, we blend the current LLM-proposed context with a randomly sampled context from history. This interpolation helps inject novelty in the curriculum while avoiding sudden changes in the curriculum.

Alternating Policy Training and Context Generation Similar to (Ma et al., 2023), we alternate between policy training and context generation. After each training phase, the agent’s performance on the current context is recorded and passed to the LLM, which conditions on a sliding window of past evaluations to generate the next context. This approach—in-context context generation—enables the LLM to implicitly reason about task difficulty and progression without gradient updates or handcrafted reward shaping.

We present the full algorithm in Algorithm 1.

5 Experimental Setup

In this section, we show experimental details and the baselines we evaluate cMALC-D against.

5.1 Experimental Details

We evaluate cMALC-D on three autonomous traffic signal control environments based on real-world data. We choose to evaluate on these datasets for three reasons:

• •

Real-World Relevance Autonomous traffic signal control has critical real-life applications in reducing congestion, emissions, and travel time. This also gives several metrics outside of reward that we can use for evaluation. More information about environment metrics can be found in Appendix A.2

• •

Well Defined Context Space Each vehicle in the traffic signal control environment is clearly defined by several variables that can be altered to generate new contexts. More details about context parameterization can be found in Appendix A.3.

• •

Multi-Agent Coordination Challenges Traffic signal control inherently involves decentralized decision-making and coordination among agents, providing a natural testbed for evaluating the scalability and generalization of our cMARL curriculum framework.

We run our experiments with the CityFlow environment, which is a realistic and efficient traffic flow simulator written in C++ (Zhang et al., 2019). It is also compatible with MARL algorithms via integration with the Gymnasium Library (Brockman, 2016). We train all policies with MAPPO (Yu et al., 2022), but any MARL algorithm will work; we choose to use MAPPO due to its efficiency compared to off-policy algorithms.

We use the Qwen2.5-7B-Instruct model (Team, 2024) as a high-level curriculum designer. To increase computational efficiency, we use the vLLM package (Kwon et al., 2023) to streamline inference and leverage activation-aware weight quantization (Lin et al., 2024) to reduce memory usage. Thus, we only require 2 RTX 2080 TI GPUs for LLM-based experiments, and 1 RTX 2080 TI GPU for non-LLM experiments, and there is a neglible difference in MARL policy training time.

For all experiments, we alternate between expanding the curriculum and training the MARL policy for 500 episodes, where each episode is 360 timesteps, resulting in 180,000 trajectories per training phase. This is done via the Centralized Training Decentralized Execution (CTDE) paradigm, where training is done with access to the global joint observation, but agents only have access to their local observations during policy execution. We reserve a held-out test set of 5 contexts and evaluate the current policy every 5 episodes using greedy action sampling. After training, we generate 10 additional random contexts to assess generalization performance in both a zero-shot setting and after a brief finetuning phase of 5 episodes. In practice, we observe minimal differences between the zero-shot and finetuned policies; therefore, we report performance metrics, such as delay time, throughput, and wait time, based on the finetuned policy. Additional details about these metrics can be found in Appendix A. All experiments are repeated across 5 random seeds to ensure statistical robustness.

5.2 Baselines

In this work, we compare cMALC-D to 5 other curriculum learning algorithms.

• •

No Curriculum: For this algorithm, we simply train MARL agents using MAPPO on a single training environment for all epochs, as would be done normally.

• •

Domain Randomization (Tzeng et al., 2020): Domain Randomization randomly generates an environment using a set of parameters and a probability distribution for each parameter at each timestep. Then, a policy is trained on a rollout from that environment at each timestep.

• •

Prioritized Level Replay (PLR) (Jiang et al., 2021): PLR is a curriculum learning algorithm that keeps track of previously generated environments (or levels) and scores them based on learning potential using the Temporal Difference Error. After evaluating each level, it decides randomly whether to replay the previously played level with the highest learning potential or create a new level.

• •

Adversarially Compounding Complexity by Editing Levels (ACCEL) (Parker-Holder et al., 2023): ACCEL is similar to PLR, except instead of training directly on previous levels with high TD-error, ACCEL maintains a population of levels with high TD-error and makes small and randomized mutations to those levels, and decides randomly whether to play on one of the population of those levels or generate a new one.

• •

Self-Paced Context Evaluation (SPACE) (Eimer et al., 2021): SPACE uses the value estimate to build a curriculum. Taking $V^{\pi}_{t}(s_{0},c_{i})$ to be the estimated total expected reward by following policy $\pi$ from starting state $s_{0}$ with context $c_{i}$ after $t$ training steps, SPACE takes $V^{\pi}_{t}(s_{0},c_{i})-V^{\pi}_{t-1}(s_{0},c_{i})$ to be the performance improvement capacity (PIC) for $c_{i}$. SPACE trains continually in environments with high PIC until it converges, sampling a new environment from the context space.

6 Results

In this section, we describe the results of cMALC-D compared to the baselines and perform ablation studies to understand individual components of the algorithm. We aim to answer three main questions: What is the generalization performance of the algorithm? How does the diversity mechanism influence context generation? What kinds of contexts are generated by the LLM curriculum?

6.1 Generalization Performance

We show the generalization performance of cMALC-D against the baseline algorithms in Tables 1(a), 1(b), and 1(c). Across all three environments, JN $1\times 3$, HZ, and JN $3\times 4$, cMALC-D consistently outperforms or matches all other curriculum strategies on the test reward and specific traffic policy metrics, such as average delay and throughput. For example, in JN $1\times 3$, it achieves the highest test reward ($29.01\pm 0.32$) and throughput ($3073.22\pm 114.06$) while reducing wait time by $2\%$ over the second-best algorithm. Similar trends hold for the HZ and JN $3\times 4$ environments.

Structured curricula are necessary to learn generalizable policies. In contrast, Domain Randomization underperforms compared to cMALC-D, often giving 3rd or 4th place results across performance metrics (e.g., 4th place in average delay in HZ with $241.79\pm 35.60$ vs. cMALC-D’s $146.96\pm 19.93$). While it occasionally yields high throughput or test rewards over other algorithms (e.g., 2nd place test reward of $27.55\pm 0.41$ in JN $1\times 3$), these gains are unreliable and highly environment-dependent. This inconsistency highlights a fundamental limitation of randomization-based strategies: while they expose agents to a wide range of environments, they do so without considering progression or context relevance. As a result, agents may struggle to learn the high-level coordination skills necessary for generalization due to rapid context switching in the curriculum.

Original context can be a useful prior, but may encourage overfitting. Training without a curriculum can yield strong performance, particularly in the JN environments, where No Curriculum frequently ranks second after cMALC-D (e.g., throughput of $3704.17\pm 147.39$ vs. cMALC-D’s $3795.46\pm 159.50$ in JN $3\times 4$). This suggests that the original context provides a good prior, enabling agents to learn basic coordination strategies. However, its effectiveness diminishes in more diverse settings (most notably in the HZ environment, where its average delay of $339.09\pm 53.28$ is worse than cMALC-D’s $146.96\pm 19.93$), where it performs significantly worse than cMALC-D and exhibits high variance even in the JN environments (e.g., test reward standard deviation of $2.44$ vs. $1.53$ in JN $3\times 4$). This drop indicates that without curriculum learning, agents may overfit to features in the original context, which limits generalizability.

LLM-based context evaluation provides a stable signal for effective curriculum learning. While some methods like ACCEL and PLR incorporate similar automatic curriculum schemes, they rely heavily on policy evaluation signals, such as value functions or generalized advantage estimates, to select and schedule tasks. While these signals can be highly effective in single-agent domains with millions of environment updates, they can be noisy or unreliable in MARL due to non-stationarity, partial observability, and inter-agent dependencies (e.g., ACCEL’s inconsistent rankings from 4th place in JN $1\times 3$ to 2nd place in HZ). On the other hand, cMALC-D’s context selection strategy promotes gradual skill acquisition that transfers well across diverse contexts. This is due to using language-based evaluations that can capture qualitative improvements that traditional metrics might overlook (demonstrated by cMALC-D’s top performance across all environments with test rewards of $29.01\pm 0.32$, $172.87\pm 1.03$, and $116.57\pm 1.53$ in JN $1\times 3$, HZ, and JN $3\times 4$, respectively).

6.2 Influence of the Diversity Mechanism

To evaluate the impact of the diversity mechanism, we compare three variants of our method: the full version with similarity-based diversity (cMALC-D), a baseline without the diversity mechanism (cMALC), and a variant that applies task arithmetic with random probability $\epsilon=0.1$ instead of using similarity checks (cMALC-$\epsilon$).

Figure 1 shows the mean test return across five held-out test tasks (rows) and three datasets (columns), with each curve averaged over five random seeds. On the easier Jinan $(1\times 3)$ dataset (left column), cMALC-D yields clear performance improvements. This is especially evident in test tasks 1, 3, and 4 (first, third, and fourth rows), where cMALC-D consistently achieves a test return around 31, outperforming the other variants by approximately two points on average.

On the Hangzhou dataset (middle column), performance across all three variants is more similar, but cMALC-D still shows consistent gains. Across all settings, the inclusion of the diversity mechanism improves sample efficiency and accelerates convergence. However, cMALC-$\epsilon$ exhibits greater instability. In particular, test tasks 1 through 4 (first four rows) show that cMALC-$\epsilon$ often suffers sharp drops in performance around 170,000 timesteps, sometimes losing much of the progress previously made.

Performance is uniformly lower on the most challenging Jinan $(3\times 4)$ dataset (right column). We attribute this to the increased traffic density, as small changes in the context can cause substantial environment changes. Nevertheless, cMALC-D remains the top-performing variant. Notably, the baseline cMALC frequently shows a decline in test reward over time (particularly visible in test tasks 1-4), which suggests mode collapse and poor generalization to test contexts, further highlighting the need for context diversity during training.

6.3 What kinds of contexts are generated?

To better understand how curricula evolve, we visualize context trajectories using heatmaps in Figures 3, 4, and 5. Across all three environments, we observe that context features tend to evolve gradually over time rather than undergoing abrupt shifts. This gradual progression facilitates a smoother learning process for the agents and supports our finding that cMALC-D outperforms Domain Randomization due to a more controlled approach of generating diverse contexts.

We also observe that in the curriculum for the JN $1\times 3$ environment, multiple context features are frequently updated simultaneously. For instance, around episode 125, changes occur simultaneously in maxSpeed and minGap, along with adjustments to vehicle length and width. A similar cluster of updates appears around episode 375, involving both maxNegAcc and usualNegAcc, and in the HZ environment near episode 450.

This raises an interesting question: *do LLMs internally form or leverage semantic relationships between features, such as coupling speed with spacing, or braking with acceleration, to propose more coherent or pedagogically aligned context transitions? * To answer this question, we construct correlation matrices between features in the cMALC-D generated curriculum in Figures 6, 7, 8.

In the JN $1\times 3$ environment, which contains fewer vehicles and simpler dynamics, the LLM identifies and exploits meaningful semantic patterns. For example, maxNegAcc (maximum negative acceleration or braking) has a weak correlation with other features, suggesting that the model implicitly recognizes the limited need for braking behaviors due to the sparsity of interactions in a small-scale scenario. In the HZ environment, vehicle width shows a similarly low correlation, despite the model not being given lane dimensions. This suggests the LLM infers structural properties of the environment, like tight lane-to-vehicle width ratios, and develops curricula that do not depend on this feature, purely from context patterns. Such reasoning allows it to generate more coherent and pedagogically effective curricula without explicit environmental parameters.

On the other hand, the LLM also captures strong correlations between semantically coupled features. For example, in the JN $3\times 4$ environment (Figure 6), the correlation between maxSpeed and minGap is 0.71, reflecting the LLM’s implicit understanding that higher speeds require larger spacing between vehicles to ensure safety. Similarly, in the HZ environment (Figure 7), usualPosAcc and maxPosAcc are correlated with a value of 0.83, indicating the LLM’s awareness that general acceleration behavior is often constrained by maximum performance limits.

7 Conclusion and Future Work

In this paper, we develop cMALC-D, an LLM-based curriculum learning algorithm for contextual MARL. Our method leverages the reasoning capabilities of LLMs to generate semantically meaningful curricula. We also introduce a novel diversity-based mechanism based on task arithmetic from continual learning to encourage exploration in the context space and avoid mode collapse. Our experiments on three real-world traffic environments show that cMALC-D enhances MARL policy generalization and sample efficiency over a variety of environment configurations.

While our formulation is based on modifying the transition function by altering context features, there are other methods to model different environment configurations. For example, in the traffic signal control setting, sensor malfunctions can cause agents to return noisy data (Yang et al., 2024). One possible way to include this is to have the LLM generate a more general context, such as a noisy environment configuration parameterized by a probability distribution. Another avenue for future work is explicitly encoding semantic feature relationships to augment self-paced curricula.

Appendix A Additional Experimental Details

A.1 Dataset Details

We use the Cityflow environment with three publicly available datasets derived from real-world data. We use the Jinan $(1\times 3)$ dataset from PressLight (Wei et al., 2019a) and the Jinan $(3\times 4)$ and Hangzhou $(4\times 4)$ datasets from CoLight (Wei et al., 2019b). Each dataset contains a roadnet.json file that gives the topology of the road network and a flow.json file that gives the vehicle routes and flow patterns. A more detailed list of dataset statistics can be found in Table 2.

A.2 Environment Details

In our experiments, each agent corresponds to a traffic light controlling an intersection. Therefore, the Jinan environments have 3 and 12 agents, respectively, while the Hangzhou environment has 16 agents. Each traffic light agent can choose from 8 possible actions, where each action represents a specific signal phase. A phase refers to a predefined combination of green and red signals assigned to incoming lanes, typically allowing certain directions of traffic (e.g., straight-through, left-turn) while restricting others. Figure 2 illustrates each phase. The observation for each intersection is composed of information about all ingoing and outgoing cars in each lane, such as their position, velocity, and lane number. Each agent has partial observability and cannot use information that is outside the local observation to make decisions. At time $t$, the global state $s_{t}$ is the concatenation of all local observations. The global reward is defined as $\lambda_{f}T_{f}+\lambda_{w}T_{w}$, where $T_{f}$ is the total time vehicles are moving, $T_{w}$ is the total time vehicles are waiting in traffic, and $\lambda_{f},\lambda_{w}$ are hyperparameters. In practice, we set $\lambda_{f}=0.033$ and $\lambda_{w}=0$.

We evaluate our traffic control policies using several real-world metrics that reflect their effectiveness. Wait time measures the total duration that vehicles remain stationary (i.e., with zero speed) during their journeys. Delay time captures the additional time each vehicle takes to reach its destination compared to the ideal travel time in the absence of any traffic signals or congestion. Finally, we calculate the throughput, which is the total number of vehicles that complete their routes before the end of the episode.

A.3 Context Parameterization Details

A.4 Hyperparameters

The MAPPO algorithm is adapted from the ePYMARL library (Papoudakis et al., 2021). We provide a full list of algorithmic environment hyperparameters in Table 4. We also use an Adam optimizer with a learning rate of $0.003$ to train the cost estimator. We also list the hyperparameters of cMALC-D and the parameters used for the LLM in Tables 5 and 6 respectively.

Appendix B Further Results

B.1 LLM Prompts