Marginal Effects for Non-Linear Prediction Functions

TL;DR

This paper introduces forward marginal effects as a flexible, model-agnostic interpretation method for non-linear prediction functions, allowing for multivariate analysis and conditional effects to better understand feature impacts.

Contribution

The paper proposes forward marginal effects and a non-linearity measure, extending marginal effects to multivariate and conditional contexts for improved interpretability of non-linear models.

Findings

Forward marginal effects outperform derivatives in interpretability.

Partitioning feature space yields more accurate conditional effects.

Non-linearity measure helps assess model complexity.

Abstract

Beta coefficients for linear regression models represent the ideal form of an interpretable feature effect. However, for non-linear models and especially generalized linear models, the estimated coefficients cannot be interpreted as a direct feature effect on the predicted outcome. Hence, marginal effects are typically used as approximations for feature effects, either in the shape of derivatives of the prediction function or forward differences in prediction due to a change in a feature value. While marginal effects are commonly used in many scientific fields, they have not yet been adopted as a model-agnostic interpretation method for machine learning models. This may stem from their inflexibility as a univariate feature effect and their inability to deal with the non-linearities found in black box models. We introduce a new class of marginal effects termed forward marginal effects. We argue to abandon derivatives in favor of better-interpretable forward differences. Furthermore, we generalize marginal effects based on forward differences to multivariate changes in feature values. To account for the non-linearity of prediction functions, we introduce a non-linearity measure for marginal effects. We argue against summarizing feature effects of a non-linear prediction function in a single metric such as the average marginal effect. Instead, we propose to partition the feature space to compute conditional average marginal effects on feature subspaces, which serve as conditional feature effect estimates.