Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrations

TL;DR

This paper introduces a novel IRL approach to learn implicit constraints from demonstrations, enabling AI agents to mimic human trade-offs in complex, constrained environments and adapt to new domains effectively.

Contribution

It proposes a new IRL method for learning implicit constraints and integrates it with a cognitive model to replicate human decision-making in AI agents.

Findings

Agents effectively learn human-like trade-offs from demonstrations.

The approach generalizes to new environments with similar constraints.

Agents outperform baseline methods in trajectory and constraint violation metrics.

Abstract

Many real-life scenarios require humans to make difficult trade-offs: do we always follow all the traffic rules or do we violate the speed limit in an emergency? These scenarios force us to evaluate the trade-off between collective norms and our own personal objectives. To create effective AI-human teams, we must equip AI agents with a model of how humans make trade-offs in complex, constrained environments. These agents will be able to mirror human behavior or to draw human attention to situations where decision making could be improved. To this end, we propose a novel inverse reinforcement learning (IRL) method for learning implicit hard and soft constraints from demonstrations, enabling agents to quickly adapt to new settings. In addition, learning soft constraints over states, actions, and state features allows agents to transfer this knowledge to new domains that share similar aspects. We then use the constraint learning method to implement a novel system architecture that leverages a cognitive model of human decision making, multi-alternative decision field theory (MDFT), to orchestrate competing objectives. We evaluate the resulting agent on trajectory length, number of violated constraints, and total reward, demonstrating that our agent architecture is both general and achieves strong performance. Thus we are able to capture and replicate human-like trade-offs from demonstrations in environments when constraints are not explicit.