Semantic Parsing to Probabilistic Programs for Situated Question Answering

TL;DR

This paper introduces P3, a novel model that interprets questions and environments as probabilistic programs, enabling effective situated question answering by handling uncertainty and leveraging background knowledge.

Contribution

The paper proposes Parsing to Probabilistic Programs (P3), a new approach that models semantic parses as probabilistic programs for improved situated question answering.

Findings

P3 outperforms classical and neural baselines on a new science diagram dataset.

The model effectively incorporates background knowledge and environmental uncertainty.

Efficient approximate inference is maintained despite the probabilistic nature.

Abstract

Situated question answering is the problem of answering questions about an environment such as an image or diagram. This problem requires jointly interpreting a question and an environment using background knowledge to select the correct answer. We present Parsing to Probabilistic Programs (P3), a novel situated question answering model that can use background knowledge and global features of the question/environment interpretation while retaining efficient approximate inference. Our key insight is to treat semantic parses as probabilistic programs that execute nondeterministically and whose possible executions represent environmental uncertainty. We evaluate our approach on a new, publicly-released data set of 5000 science diagram questions, outperforming several competitive classical and neural baselines.