Distilling Task-specific Logical Rules from Large Pre-trained Models

TL;DR

This paper introduces STREAM, a framework that distills task-specific logical rules from large pre-trained models using prompt-based techniques, significantly improving named entity tagging performance without extensive human-annotated seed rules.

Contribution

The novel framework STREAM leverages prompt-based language models to automatically generate high-quality seed rules, reducing human effort and enhancing rule learning from large pre-trained models.

Findings

Significant improvements over previous methods on named entity tagging benchmarks.

Effective use of prompt templates to generate initial seed rules.

Demonstrated ability to learn task-specific logical rules with less human annotation.

Abstract

Logical rules, both transferable and explainable, are widely used as weakly supervised signals for many downstream tasks such as named entity tagging. To reduce the human effort of writing rules, previous researchers adopt an iterative approach to automatically learn logical rules from several seed rules. However, obtaining more seed rules can only be accomplished by extra human annotation with heavy costs. Limited by the size and quality of the seed rules, the model performance of previous systems is bounded. In this paper, we develop a novel framework STREAM to distill task-specific logical rules from large pre-trained models. Specifically, we borrow recent prompt-based language models as the knowledge expert to yield initial seed rules, and based on the formed high-quality instance pool that acts as an intermediary role, we keep teaching the expert to fit our task and learning task-specific logical rules. Experiments on three public named entity tagging benchmarks demonstrate the effectiveness of our proposed framework. With several predefined prompt templates, our system has gained significant improvements over previous state-of-the-art methods.