A Monte Carlo comparison of categorical tests of independence

TL;DR

This study compares chi-squared and G-squared tests for categorical independence, recommending chi-squared for small samples and permutation G-squared for conditional independence, based on extensive Monte Carlo simulations.

Contribution

It provides the first comprehensive Monte Carlo comparison of these tests, clarifying their applicability and limitations in different sample size scenarios.

Findings

Chi-squared and G-squared tests are similar in large samples.

Chi-squared test is recommended for small samples regardless of zero frequencies.

Permutation G-squared is suggested for conditional independence testing.

Abstract

The $X^2$ and $G^2$ tests are the most frequently applied tests for testing the independence of two categorical variables. However, no one, to the best of our knowledge has compared them, extensively, and ultimately answer the question of which to use and when. Further, their applicability in cases with zero frequencies has been debated and (non parametric) permutation tests are suggested. In this work we perform extensive Monte Carlo simulation studies attempting to answer both aforementioned points. As expected, in large sample sized cases ($>1,000$) the $X^2$ and $G^2$ are indistinguishable. In the small sample sized cases ($\leq 1,000$) though, we provide strong evidence supporting the use of the $X^2$ test regardless of zero frequencies for the case of unconditional independence. Also, we suggest the use of the permutation based $G^2$ test for testing conditional independence, at the cost of being computationally more expensive. The $G^2$ test exhibited inferior performance and its use should be limited.