Exploiting Field Dependencies for Learning on Categorical Data

TL;DR

This paper introduces a meta-learning based method that explicitly models and refines dependencies between categorical data fields, leading to improved performance over existing approaches.

Contribution

It proposes a novel global and local field dependency modeling approach using meta-learning, which is refined without labels and integrated with supervised embedding updates.

Findings

Outperforms state-of-the-art methods on six datasets

Effective modeling of field dependencies improves predictive accuracy

Ablation studies validate the importance of dependency refinement

Abstract

Traditional approaches for learning on categorical data underexploit the dependencies between columns (\aka fields) in a dataset because they rely on the embedding of data points driven alone by the classification/regression loss. In contrast, we propose a novel method for learning on categorical data with the goal of exploiting dependencies between fields. Instead of modelling statistics of features globally (i.e., by the covariance matrix of features), we learn a global field dependency matrix that captures dependencies between fields and then we refine the global field dependency matrix at the instance-wise level with different weights (so-called local dependency modelling) w.r.t. each field to improve the modelling of the field dependencies. Our algorithm exploits the meta-learning paradigm, i.e., the dependency matrices are refined in the inner loop of the meta-learning algorithm without the use of labels, whereas the outer loop intertwines the updates of the embedding matrix (the matrix performing projection) and global dependency matrix in a supervised fashion (with the use of labels). Our method is simple yet it outperforms several state-of-the-art methods on six popular dataset benchmarks. Detailed ablation studies provide additional insights into our method.