Efficient Induction of Language Models Via Probabilistic Concept Formation

TL;DR

This paper introduces three extensions to the Cobweb system for incremental, probabilistic language model induction, enabling online processing of sequential language data and improving synonym grouping and homonym separation.

Contribution

The paper develops Word, Leaf, and Path variants of Cobweb that encode language context and update hierarchies incrementally, adapting a taxonomic approach to sequential language learning.

Findings

Effective synonym grouping demonstrated

Homonyms are kept apart successfully

Training efficiency is improved

Abstract

This paper presents a novel approach to the acquisition of language models from corpora. The framework builds on Cobweb, an early system for constructing taxonomic hierarchies of probabilistic concepts that used a tabular, attribute-value encoding of training cases and concepts, making it unsuitable for sequential input like language. In response, we explore three new extensions to Cobweb -- the Word, Leaf, and Path variants. These systems encode each training case as an anchor word and surrounding context words, and they store probabilistic descriptions of concepts as distributions over anchor and context information. As in the original Cobweb, a performance element sorts a new instance downward through the hierarchy and uses the final node to predict missing features. Learning is interleaved with performance, updating concept probabilities and hierarchy structure as classification occurs. Thus, the new approaches process training cases in an incremental, online manner that it very different from most methods for statistical language learning. We examine how well the three variants place synonyms together and keep homonyms apart, their ability to recall synonyms as a function of training set size, and their training efficiency. Finally, we discuss related work on incremental learning and directions for further research.