A Free Lunch with Influence Functions? Improving Neural Network Estimates with Concepts from Semiparametric Statistics

TL;DR

This paper investigates how influence functions can enhance neural network estimates by improving robustness and inference without additional data or retraining, through a new method called 'MultiNet' and variants 'MultiStep'.

Contribution

It introduces a neural network method 'MultiNet' that leverages influence functions for improved estimation, robustness, and inference, demonstrating dataset-dependent benefits.

Findings

Influence functions can improve neural network estimates without extra data.

The effectiveness of methods depends on the data generating process.

Practitioners should validate their results with multiple estimators.

Abstract

Parameter estimation in empirical fields is usually undertaken using parametric models, and such models readily facilitate statistical inference. Unfortunately, they are unlikely to be sufficiently flexible to be able to adequately model real-world phenomena, and may yield biased estimates. Conversely, non-parametric approaches are flexible but do not readily facilitate statistical inference and may still exhibit residual bias. We explore the potential for Influence Functions (IFs) to (a) improve initial estimators without needing more data (b) increase model robustness and (c) facilitate statistical inference. We begin with a broad introduction to IFs, and propose a neural network method 'MultiNet', which seeks the diversity of an ensemble using a single architecture. We also introduce variants on the IF update step which we call 'MultiStep', and provide a comprehensive evaluation of different approaches. The improvements are found to be dataset dependent, indicating an interaction between the methods used and nature of the data generating process. Our experiments highlight the need for practitioners to check the consistency of their findings, potentially by undertaking multiple analyses with different combinations of estimators. We also show that it is possible to improve existing neural networks for `free', without needing more data, and without needing to retrain them.