Open Rule Induction

TL;DR

This paper introduces open rule induction, leveraging language models to generate flexible, expressive rules beyond knowledge bases, and presents the Orion system for automatic rule mining that improves downstream task performance.

Contribution

It defines the open rule induction problem and proposes the Orion system to automatically mine open rules from language models, surpassing previous KB-based methods.

Findings

Open rules outperform manually annotated rules in relation extraction.

The Orion system effectively mines high-quality open rules from LMs.

Open rule induction enhances rule expressiveness and generalization.

Abstract

Rules have a number of desirable properties. It is easy to understand, infer new knowledge, and communicate with other inference systems. One weakness of the previous rule induction systems is that they only find rules within a knowledge base (KB) and therefore cannot generalize to more open and complex real-world rules. Recently, the language model (LM)-based rule generation are proposed to enhance the expressive power of the rules. In this paper, we revisit the differences between KB-based rule induction and LM-based rule generation. We argue that, while KB-based methods inducted rules by discovering data commonalities, the current LM-based methods are "learning rules from rules". This limits these methods to only produce "canned" rules whose patterns are constrained by the annotated rules, while discarding the rich expressive power of LMs for free text. Therefore, in this paper, we propose the open rule induction problem, which aims to induce open rules utilizing the knowledge in LMs. Besides, we propose the Orion (\underline{o}pen \underline{r}ule \underline{i}nducti\underline{on}) system to automatically mine open rules from LMs without supervision of annotated rules. We conducted extensive experiments to verify the quality and quantity of the inducted open rules. Surprisingly, when applying the open rules in downstream tasks (i.e. relation extraction), these automatically inducted rules even outperformed the manually annotated rules.