Safe adaptation in multiagent competition

TL;DR

This paper introduces a safe adaptation method for multiagent competition that trains ego-agents against regularized opponent models, enhancing robustness and reducing exploitability in dynamic environments.

Contribution

The paper proposes a novel safe adaptation approach using regularized opponent models to prevent overfitting and improve robustness in multiagent competitive scenarios.

Findings

Effective adaptation to specific opponents

Maintains low exploitability to other opponents

Improves robustness of ego-agent policies

Abstract

Achieving the capability of adapting to ever-changing environments is a critical step towards building fully autonomous robots that operate safely in complicated scenarios. In multiagent competitive scenarios, agents may have to adapt to new opponents with previously unseen behaviors by learning from the interaction experiences between the ego-agent and the opponent. However, this adaptation is susceptible to opponent exploitation. As the ego-agent updates its own behavior to exploit the opponent, its own behavior could become more exploitable as a result of overfitting to this specific opponent's behavior. To overcome this difficulty, we developed a safe adaptation approach in which the ego-agent is trained against a regularized opponent model, which effectively avoids overfitting and consequently improves the robustness of the ego-agent's policy. We evaluated our approach in the Mujoco domain with two competing agents. The experiment results suggest that our approach effectively achieves both adaptation to the specific opponent that the ego-agent is interacting with and maintaining low exploitability to other possible opponent exploitation.