Robust agents learn causal world models
Jonathan Richens, Tom Everitt
TL;DR
This paper demonstrates that to achieve robust generalization under distributional shifts, agents must learn approximate causal models of their environment, with optimal agents converging to the true causal model.
Contribution
It proves that learning an approximate causal model is necessary for agents to generalize robustly, linking causal reasoning to regret bounds under distributional shifts.
Findings
Agents satisfying regret bounds must learn causal models
Optimal agents' causal models converge to the true model
Implications for transfer learning and causal inference
Abstract
It has long been hypothesised that causal reasoning plays a fundamental role in robust and general intelligence. However, it is not known if agents must learn causal models in order to generalise to new domains, or if other inductive biases are sufficient. We answer this question, showing that any agent capable of satisfying a regret bound under a large set of distributional shifts must have learned an approximate causal model of the data generating process, which converges to the true causal model for optimal agents. We discuss the implications of this result for several research areas including transfer learning and causal inference.
Peer Reviews
Decision·ICLR 2024 oral
This paper makes an original and significant theoretical contribution by formally establishing a fundamental connection between causal learning and generalisation under distribution shifts. ## Originality: * They provide a proof for showing that an agent that is sufficiently adaptive has learned a causal model of the environment. This is an impressive achievement and a stronger statement than the one stated by good regulator theorem (which as the authors have cited, has been misunderstood and mi
- As the authors acknowledge, the results are mainly theoretical. Even a minimal empirical validation of the key insights would strengthen the paper. For example it would be great even if you turn the informal overview (appendix C) into a simple simulation example rather than remain a thought experiment. - The scope is currently limited to unmediated decision tasks. Extending the results to broader RL settings would increase applicability (although I acknowledge that seems significantly more cha
This paper is a gem. The theoretical analysis is simple and clear, the implications are broad and powerful.
The only weakness, in my opinion, is that the statement of the result in the introduction felt pretty slippery. (See detailed comments below.) All of this was satisfyingly resolved, but I do think the paper would benefit from an effort to sharpen that first section. Details comments: - Please define these: "distributional shifts" "distributionally shifted environments" "target domains" "causal modelling and transfer learning" - " used to derive out results" typo - "Our analysis focuses on di
1. They propose theoretical results connecting decision making and causal structure learning. As suggested by their results, a robust enough agent should always learn the causal structure. 2. The limitation for learning causal structure can be transferred to limitation of robust decision making by their results. 3. Their result gives an example about inferring causal structure when only one variable is observed under each intervention.
1. They do not conduct an experiment for justifying their results. 2. Their results can only be applied to a small range of scenarios, where we need to reach small regret for all mixture of local interventions. However, most applicable tasks, such as transfer learning, only consider interventions on a subset of variables. 3. There are some spelling mistakes in their text, and some usage of notations are unclear in their text and proof.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsBayesian Modeling and Causal Inference
MethodsSparse Evolutionary Training