# Using Background Knowledge to Rank Itemsets

## TL;DR

This paper introduces methods to incorporate various types of background knowledge into the ranking of itemsets using a maximum entropy approach, improving the assessment of pattern interestingness in data mining.

## Contribution

It presents efficient techniques for infusing diverse background knowledge into itemset evaluation, extending beyond simple independence models, with polynomial-time solutions.

## Key findings

- More sophisticated models better fit the data.
- Using additional background knowledge improves frequency prediction.
- Efficient polynomial-time algorithms are developed for maximum entropy models.

## Abstract

Assessing the quality of discovered results is an important open problem in data mining. Such assessment is particularly vital when mining itemsets, since commonly many of the discovered patterns can be easily explained by background knowledge. The simplest approach to screen uninteresting patterns is to compare the observed frequency against the independence model. Since the parameters for the independence model are the column margins, we can view such screening as a way of using the column margins as background knowledge. In this paper we study techniques for more flexible approaches for infusing background knowledge. Namely, we show that we can efficiently use additional knowledge such as row margins, lazarus counts, and bounds of ones. We demonstrate that these statistics describe forms of data that occur in practice and have been studied in data mining. To infuse the information efficiently we use a maximum entropy approach. In its general setting, solving a maximum entropy model is infeasible, but we demonstrate that for our setting it can be solved in polynomial time. Experiments show that more sophisticated models fit the data better and that using more information improves the frequency prediction of itemsets.