What If: Generating Code to Answer Simulation Questions

TL;DR

This paper introduces a neural program synthesis method that generates and executes code in a domain-specific language to answer complex chemistry and biology process questions, significantly improving accuracy over existing methods.

Contribution

It presents a novel dataset, a domain-specific language for process representation, and a reinforcement learning approach with a semantic reward for improved simulation-based question answering.

Findings

Achieved 88% accuracy on simulation questions.

Outperformed state-of-the-art neural program synthesis methods.

Outperformed end-to-end text-based approaches.

Abstract

Many texts, especially in chemistry and biology, describe complex processes. We focus on texts that describe a chemical reaction process and questions that ask about the process's outcome under different environmental conditions. To answer questions about such processes, one needs to understand the interactions between the different entities involved in the process and to simulate their state transitions during the process execution under different conditions. A state transition is defined as the memory modification the program does to the variables during the execution. We hypothesize that generating code and executing it to simulate the process will allow answering such questions. We, therefore, define a domain-specific language (DSL) to represent processes. We contribute to the community a unique dataset curated by chemists and annotated by computer scientists. The dataset is composed of process texts, simulation questions, and their corresponding computer codes represented by the DSL.We propose a neural program synthesis approach based on reinforcement learning with a novel state-transition semantic reward. The novel reward is based on the run-time semantic similarity between the predicted code and the reference code. This allows simulating complex process transitions and thus answering simulation questions. Our approach yields a significant boost in accuracy for simulation questions: 88\% accuracy as opposed to 83\% accuracy of the state-of-the-art neural program synthesis approaches and 54\% accuracy of state-of-the-art end-to-end text-based approaches.