Knowledge Authoring and Question Answering with KALM

TL;DR

KALM is a rule-based system that enables users to easily author and query large knowledge bases via text, achieving higher accuracy than existing systems in knowledge representation and question answering.

Contribution

The paper introduces KALM, a novel rule-based system that simplifies knowledge authoring and querying, improving accuracy over current state-of-the-art methods.

Findings

KALM achieved superior accuracy in knowledge authoring.

KALM outperformed existing systems in question answering.

The system reduces the need for qualified knowledge engineers.

Abstract

Knowledge representation and reasoning (KRR) is one of the key areas in artificial intelligence (AI) field. It is intended to represent the world knowledge in formal languages (e.g., Prolog, SPARQL) and then enhance the expert systems to perform querying and inference tasks. Currently, constructing large scale knowledge bases (KBs) with high quality is prohibited by the fact that the construction process requires many qualified knowledge engineers who not only understand the domain-specific knowledge but also have sufficient skills in knowledge representation. Unfortunately, qualified knowledge engineers are in short supply. Therefore, it would be very useful to build a tool that allows the user to construct and query the KB simply via text. Although there is a number of systems developed for knowledge extraction and question answering, they mainly fail in that these system don't achieve high enough accuracy whereas KRR is highly sensitive to erroneous data. In this thesis proposal, I will present Knowledge Authoring Logic Machine (KALM), a rule-based system which allows the user to author knowledge and query the KB in text. The experimental results show that KALM achieved superior accuracy in knowledge authoring and question answering as compared to the state-of-the-art systems.

Full text

1 Introduction

Knowledge representation and reasoning (KRR) is the process of representing the domain knowledge in formal languages (e.g., SPARQL, Prolog) such that it can be used by expert systems to execute querying and reasoning services. KRR have been applied in many fields including financial regulations, medical diagnosis, laws, and so on. One major obstacle in KRR is the creation of large-scale knowledge bases with high quality. For one thing, this requires the knowledge engineers (KEs) not only to have the background knowledge in a certain domain but have enough skills in knowledge representation as well. Unfortunately, qualified KEs are also in short supply. Therefore, it would be useful to build a tool that allows the domain experts without any background in logic to construct and query the knowledge base simply from text.

Controlled natural languages (CNLs) [12] were developed as a technology that achieves this goal. CNLs are designed based on natural languages (NLs) but with restricted syntax and interpretation rules that determine the unique meaning of the sentence. Representative CNLs include Attempto Controlled English [5] and PENG [19]. Each CNL is developed with a language parser which translates the English sentences into an intermediate structure, discourse representation structure (DRS) [11]. Based on the DRS structure, the language parsers further translate the DRS into the corresponding logical representations, e.g., Answer Set Programming (ASP) [9] programs. One main issue with the aforementioned CNLs is that the systems do not provide enough background knowledge to preserve semantic equivalences of sentences that represent the same meaning but are expressed via different linguistic structures. For instance, the sentences Mary buys a car and Mary makes a purchase of a car are translated into different logical representations by the current CNL parsers. As a result, if the user ask a question who is a buyer of a car, these systems will fail to find the answer.

In this thesis proposal, I will present KALM [8, 7], a system for knowledge authoring and question answering. KALM is superior to the current CNL systems in that KALM has a complex frame-semantic parser which can standardize the semantics of the sentences that express the same meaning via different linguistic structures. The frame-semantic parser is built based on FrameNet [10] and BabelNet [16] where FrameNet is used to capture the meaning of the sentence and BabelNet [16] is used to disambiguate the meaning of the extracted entities from the sentence. Experiment results show that KALM achieves superior accuracy in knowledge authoring and question answering as compared to the state-of-the-art systems.

The rest parts are organized as follows: Section 2 presents the KALM architecture, Section 3 presents KALM-QA, the question answering part of KALM, Section 4 shows the evaluation results, Section 5 discusses the related works, Section 6 shows the future work beyond the thesis, and Section 7 concludes the paper.

2 The KALM Architecture

Figure 1 shows the architecture of KALM which translates a CNL sentence to the corresponding logical representations, unique logical representations (ULR).

Attempto Parsing Engine. The input sentences are CNL sentences based on ACE grammar.111http://attempto.ifi.uzh.ch/site/docs/syntaxreport.html KALM starts with parsing the input sentence using ACE Parser222https://github.com/Attempto/APE and generates the DRS structure [6] which captures the syntactic information of the sentences.

Frame Parser. KALM performs frame-based parsing based on the DRS and produces a set of frames that represent the semantic relations a sentence implies. A frame [4] represents a semantic relation of a set of entities where each plays a particular role in the frame relation. We have designed a frame ontology, called FrameOnt, which is based on the frames in FrameNet [10] and encoded as a Prolog fact. For instance, the Commerce_Buy frame is shown below:

fp(Commerce_Buy,[
   role(Buyer,[bn:00014332n],[]),
   role(Seller,[bn:00053479n],[]),
   role(Goods,[bn:00006126n,bn:00021045n],[]),
   role(Recipient,[bn:00066495n],[]),
   role(Money,[bn:00017803n],[currency])]).

In each role-term, the first argument is the name of the role and the second is a list of role meanings represented via BabelNet synset IDs [16]. The third argument of a role-term is a list of constraints on that role. For instance, the sentence Mary buys a car implies the Commerce_Buy frame where Mary is the Buyer and car is the Goods. To extract a frame instance from a given CNL sentence, KALM uses logical valence patterns (lvps) which are learned via structural learning. An example of the lvp is shown below:

lvp(buy,v,Commerce_Buy, [
  pattern(Buyer,verb->subject,required),
  pattern(Goods,verb->object,required),
  pattern(Recipient,verb->pp[for]->dep,optnl),
  pattern(Money,verb->pp[for]->dep,optnl),
  pattern(Seller,verb->pp[from]->dep,optnl)]).

The first three arguments of an lvp-fact identify the lexical unit, its part of speech, and the frame. The fourth argument is a set of pattern-terms, each having three parts: the name of a role, a grammatical pattern, and the required/optional flag. The grammatical pattern determines the grammatical context in which the lexical unit, a role, and a role-filler word can appear in that frame. Each grammatical pattern is captured by a parsing rule (a Prolog rule) that can be used to extract appropriate role-filler words based on the APE parses.

Role-filler Disambiguation. Based on the extracted frame instance, the role-filler disambiguation module disambiguates the meaning of each role-filler word for the corresponding frame role a BabelNet Synset ID. A complex algorithm [8] was proposed to measure the semantic similarity between a candidate BabelNet synset that contains the role-filler word and the frame-role synset. The algorithm also has optimizations that improve the efficiency of the algorithm e.g., priority-based search, caching, and so on. In addition to disambiguating the meaning of the role-fillers, this module is also used to prune the extracted frame instances where the role-filler word and the frame role are semantically incompatible.

Constructing ULR. The extracted frame instances are translated into the corresponding logical representations, unique logical representation (ULR). Examples can be found in reference [8].

3 KALM-QA for Question Answering

Based on KALM, KALM-QA [7] is developed for question answering. Figure 2 shows the KALM-QA architecture. KALM-QA shares the same components with KALM for syntactic parsing, frame-based parsing and role-filler disambiguation. Different from KALM, KALM-QA translates the questions to unique logical representation for queries (ULRQ), which are used to query the authored knowledge base.

4 Evaluations

This section provides a summary of the evaluation of KALM and KALM-QA, where KALM is evaluated for knowledge authoring and KALM-QA is evaluated for question answering. We have created a total of 50 logical frames, mostly derived from FrameNet but also some that FrameNet is missing (like Restaurant, Human_Gender) for representing the meaning of English sentences. Based on the 50 frames, we have manually constructed 250 sentences that are adapted from FrameNet exemplar sentences and evaluate these sentences on KALM, SEMAFOR, SLING, and Stanford KBP system.

The evaluation is based on the following metrics:

[TABLE]

The results are shown as follow:

** **KALM:

239 sentences are FrSynC (95.6%), 248 sentences are FrC ($>$ 99%), and 2 sentences are Wrong ($<$ 1%). Note that FrSynC applies only to KALM, since none of the comparison systems can disambiguate the senses of the extracted entities.

** **SEMAFOR:

parses 236 sentences out of the 250 test sentences, where 59 sentences are FrC (25%), 44 sentences are PFrC (18.6%), and 133 sentences are Wrong (56.4%).

** **SLING:

parses 233 sentences, where 98 sentences are FrC (42.1%), 63 are PFrC (27%), and 72 sentences are Wrong (30.9%).

** **Stanford KBP:

parses 26 sentences, where 14 sentences are FrC (53.8%), 10 sentences are PrC (38.5%), and 2 sentences are Wrong (7.7%).

The differences between KALM and other systems are listed in order. First, none of the other systems do disambiguation or attempt to find synsets for role-fillers, so in this aspect KALM does more and is better attuned to the task of knowledge authoring. Second, none of these systems can explain their results, nor do they provide ways to analyze and correct errors. Third, KALM achieves an accuracy of 95.6%—much higher than other systems.

For KALM-QA, we evaluate it on two datasets. The first dataset is manually constructed general questions based on the 50 logical frames. KALM-QA achieves an accuracy of 95% for parsing the queries. The second dataset we use is MetaQA dataset [21], which contains contains almost 29,000 test questions and over 260,000 training questions. KALM-QA achieves 100% accuracy—much higher than the state-of-the-art machine learning approach [21]. Details of the evaluations can be found in [8] and [7].

5 Related Works

As is described in Section 1, CNL systems were proposed as the technology for knowledge representation and reasoning. Related works also include knowledge extraction tools, e.g., OpenIE [2], SEMAFOR [3], SLING [17], and Standford KBP system [14]. These knowledge extraction tools are designed to extract semantic relations from English sentences that capture the meaning. The limitations of these tools are two-fold: first, they lack sufficient accuracy to extract the correct semantic relations and entities while KRR is very sensitive to incorrect data; second, these systems are not able to map the semantic relations to logical forms and therefore not capable of doing KRR. Other related works include the question answering frameworks, e.g., Memory Network [15], Variational Reasoning Network [21], ATHENA [18], PowerAqua [13]. The first two belong to end-to-end learning approaches based on machine learning models. The last two systems have implemented semantic parsers which translate natural language sentences into intermediate query languages and then query the knowledge base to get the answers. For the machine learning based approaches, the results are not explainable. Besides, their accuracy is not high enough to provide correct answers. For ATHENA and PowerAqua, these systems perform question answering based on a priori knowledge bases. Therefore, they do not support knowledge authoring while KALM is able to support both knowledge authoring and question answering.

6 Future Work Beyond The Thesis

This section discusses the future work beyond the thesis: (1) enhancing KALM to author rules, and (2) supporting time reasoning.

Authoring Rules from CNL. There are two research problems with rules. The first problem is the standardization of rules parses that express the same information but via different syntactic forms or using different expressions. Suppose the knowledge base contains sentences like: (1) if a person buys a car then the person owns the car, (2) every person who is a purchaser of a car is an owner of the car, (3) if a car is bought by a person then the person possesses the car. All the above sentences represent rules and express exactly the same meaning. However, KALM’s current syntactic parser will represent them in different DRSs and therefore not being able to map them into the same logical form. The second problem involves the recognition and representation of different types of rules in logic. For instance, defeasible rules are very common in text. However, this type of rules cannot be handled by first order logic. We believe defeasible logic [20] is a good fit.

Time Reasoning. Time-related information is a crucial part of human knowledge, but semantic parsing that takes the time into account is rather hard. However, we can develop a CNL that would incorporate enough time related idioms to be useful in a number of domains of discourse (e.g., tax law). Time can then be added to DRSs and incorporated into our frame based approach down to the very level of the logical facts into which sentences will be translated. This time information can be represented either via special time-aware relations among events (e.g., before, after, causality, triggering) or using a reserved argument to represent time in each fluent.

7 Conclusions

This thesis proposal provides an overview of KALM, a system for knowledge authoring. In addition, it introduces KALM-QA, the question answering part of KALM. Experimental results show that both KALM and KALM-QA achieve superior accuracy as compared to the state-of-the-art systems.