An Empirical Investigation into Deep and Shallow Rule Learning

TL;DR

This paper empirically compares deep and shallow rule learning, demonstrating that deep rule networks outperform shallow ones on artificial and real-world data, suggesting structured, layered representations are more effective.

Contribution

It introduces a uniform algorithm for deep and shallow rule learning and provides empirical evidence of the superiority of deep rule networks over shallow ones.

Findings

Deep rule networks outperform shallow networks.

Structured, layered representations facilitate better learning.

Empirical results on benchmark datasets support the effectiveness of deep rule learning.

Abstract

Inductive rule learning is arguably among the most traditional paradigms in machine learning. Although we have seen considerable progress over the years in learning rule-based theories, all state-of-the-art learners still learn descriptions that directly relate the input features to the target concept. In the simplest case, concept learning, this is a disjunctive normal form (DNF) description of the positive class. While it is clear that this is sufficient from a logical point of view because every logical expression can be reduced to an equivalent DNF expression, it could nevertheless be the case that more structured representations, which form deep theories by forming intermediate concepts, could be easier to learn, in very much the same way as deep neural networks are able to outperform shallow networks, even though the latter are also universal function approximators. In this paper, we empirically compare deep and shallow rule learning with a uniform general algorithm, which relies on greedy mini-batch based optimization. Our experiments on both artificial and real-world benchmark data indicate that deep rule networks outperform shallow networks.