Corpus-Driven Knowledge Acquisition for Discourse Analysis

TL;DR

This paper explores how machine learning techniques can leverage large text corpora to automatically acquire knowledge for discourse analysis and information extraction, reducing manual effort and improving scalability across domains.

Contribution

It demonstrates the use of ML algorithms to capture implicit domain knowledge from text corpora for discourse analysis and IE, bypassing the need for extensive hand-coded heuristics.

Findings

ML supports knowledge acquisition at higher language analysis levels

Reduces manual effort in porting IE systems to new domains

Enhances scalability and portability of information extraction

Abstract

The availability of large on-line text corpora provides a natural and promising bridge between the worlds of natural language processing (NLP) and machine learning (ML). In recent years, the NLP community has been aggressively investigating statistical techniques to drive part-of-speech taggers, but application-specific text corpora can be used to drive knowledge acquisition at much higher levels as well. In this paper we will show how ML techniques can be used to support knowledge acquisition for information extraction systems. It is often very difficult to specify an explicit domain model for many information extraction applications, and it is always labor intensive to implement hand-coded heuristics for each new domain. We have discovered that it is nevertheless possible to use ML algorithms in order to capture knowledge that is only implicitly present in a representative text corpus. Our work addresses issues traditionally associated with discourse analysis and intersentential inference generation, and demonstrates the utility of ML algorithms at this higher level of language analysis. The benefits of our work address the portability and scalability of information extraction (IE) technologies. When hand-coded heuristics are used to manage discourse analysis in an information extraction system, months of programming effort are easily needed to port a successful IE system to a new domain. We will show how ML algorithms can reduce this