A Neural-Symbolic Framework for Mental Simulation

TL;DR

This paper introduces a neural-symbolic framework combining neural networks, memory, and physics simulation for continuous environment understanding and mental simulation, adaptable through lifelong meta-learning.

Contribution

It presents a novel neural-symbolic system integrating capsules, memory, physics, and meta-learning for lifelong environment modeling and simulation.

Findings

The capsule network learns new semantics with few-shot learning.

The framework can predict physical properties and simulate interactions.

It enables tasks like navigation, sorting, and game environment simulation.

Abstract

We present a neural-symbolic framework for observing the environment and continuously learning visual semantics and intuitive physics to reproduce them in an interactive simulation. The framework consists of five parts, a neural-symbolic hybrid network based on capsules for inverse graphics, an episodic memory to store observations, an interaction network for intuitive physics, a meta-learning agent that continuously improves the framework and a querying language that acts as the framework's interface for simulation. By means of lifelong meta-learning, the capsule network is expanded and trained continuously, in order to better adapt to its environment with each iteration. This enables it to learn new semantics using a few-shot approach and with minimal input from an oracle over its lifetime. From what it learned through observation, the part for intuitive physics infers all the required physical properties of the objects in a scene, enabling predictions. Finally, a custom query language ties all parts together, which allows to perform various mental simulation tasks, such as navigation, sorting and simulation of a game environment, with which we illustrate the potential of our novel approach.