Skill Expansion and Composition in Parameter Space
Tenglong Liu, Jianxiong Li, Yinan Zheng, Haoyi Niu, Yixing Lan, Xin, Xu, Xianyuan Zhan
TL;DR
This paper introduces PSEC, a framework for efficient skill expansion and composition in autonomous agents using parameter-efficient finetuning and skill merging, enabling better reuse of prior knowledge for new challenges.
Contribution
The paper proposes a novel framework that combines skill primitives as LoRA modules and dynamic skill activation for efficient skill expansion and composition in agents.
Findings
PSEC outperforms existing methods on D4RL, DSRL, and DeepMind Control Suite benchmarks.
It demonstrates effective skill reuse and composition for diverse tasks.
The framework enables continual skill evolution with high training efficiency.
Abstract
Humans excel at reusing prior knowledge to address new challenges and developing skills while solving problems. This paradigm becomes increasingly popular in the development of autonomous agents, as it develops systems that can self-evolve in response to new challenges like human beings. However, previous methods suffer from limited training efficiency when expanding new skills and fail to fully leverage prior knowledge to facilitate new task learning. In this paper, we propose Parametric Skill Expansion and Composition (PSEC), a new framework designed to iteratively evolve the agents' capabilities and efficiently address new challenges by maintaining a manageable skill library. This library can progressively integrate skill primitives as plug-and-play Low-Rank Adaptation (LoRA) modules in parameter-efficient finetuning, facilitating efficient and flexible skill expansion. This…
Peer Reviews
Decision·ICLR 2025 Poster
The paper is easy to follow and clearly written. The experiment suite is diverse and proves the ability of SPEC to learn and compose skills. Regarding originality and significance I particularly found interesting the usage if diffusion models to adjust the weight levels of the compositions of skills and the study for skill composiiton on the differenc spaces (parameter, noise and action spaces). It made very clear the thought process of the authors to design the framework.
My biggest concern with this paper is that this is not the first paper proposeing using LoRa for multiple task leanring, e.g. [1,2] and while SPEC is clearly different from previous existing approaches, some level of comparison, theorethical or empirical would be greatly benefitial. Specifially, at present is difficult to discern what are the novel components within SPEC wrt previous skill learning frameworks. It would be also good if authors could provide their though contrasting SPEC with ex
o This paper presents an interesting idea of how to compose skills in the parameter space, and provide a clear categorization of skill composition: parameter-level, noise-level, and action level. This paper shows a way where neural network-based skills can be composed at the parameter space level, instead of composing at the action output level. o In the long run, the proposed method can open up the possibility for efficient skill learning based on large models in decision-making domai
o While the general motivation in the first paragraph makes sense, the example needs to be justified a bit better From online statistics, the time difference between child learning to walk and learning to stand without support is on average 2-2.5 months. Not everyone would call this "rapid" (Ln 32) o While the idea of parameter-level composition is interesting and also introduces fewer bottlenecks compared to methods like action level, the argument of parameter-level composition being s
The key features of PSEC include efficient skill learning and storage utilizing LoRA, direct skill synthesis in parameter space, adaptability through a context-aware module, and the ability for continuous skill expansion. Its effectiveness has been validated through various experiments, demonstrating versatility across different scenarios.
* While the strong assumption about the expressiveness of the pre-trained policy and the scalability issue of the skill library appear to be weaknesses, these seem to have been addressed in the paper's appendix. * A potential limitation of the authors' proposed framework is its applicability only to environments where data is available. This could be considered a weakness of the paper. It is conceivable that if the framework could be extended to unseen tasks through transfer learning or fast ada
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEducation Systems and Policy · Labor market dynamics and wage inequality
MethodsLib