Natural language understanding for logical games

TL;DR

This paper presents a system that automatically solves logical puzzles expressed in natural language by translating them into first-order logic and using model finding, providing explanations for its answers, with an overall accuracy of 80.89%.

Contribution

The paper introduces a novel approach combining natural language parsing, logical reasoning, and explainability for solving and explaining natural language puzzles.

Findings

Achieved 80.89% success rate on various puzzles.

Enabled the system to provide explanations via graphical proof representations.

Successfully handled 382 knights and knaves puzzles.

Abstract

We developed a system able to automatically solve logical puzzles in natural language. Our solution is composed by a parser and an inference module. The parser translates the text into first order logic (FOL), while the MACE4 model finder is used to compute the models of the given FOL theory. We also empower our software agent with the capability to provide Yes/No answers to natural language questions related to each puzzle. Moreover, in line with Explainalbe Artificial Intelligence (XAI), the agent can back its answer, providing a graphical representation of the proof. The advantage of using reasoning for Natural Language Understanding (NLU) instead of Machine learning is that the user can obtain an explanation of the reasoning chain. We illustrate how the system performs on various types of natural language puzzles, including 382 knights and knaves puzzles. These features together with the overall performance rate of 80.89\% makes the proposed solution an improvement upon similar solvers for natural language understanding in the puzzles domain.