Estimation under Model Misspecification with Fake Features

TL;DR

This paper introduces a framework for understanding estimation errors under model misspecification, especially focusing on the role of fake features, and demonstrates that fake features can improve estimation despite not being correlated with true features.

Contribution

It presents a novel framework that jointly considers fake features and covariance errors, revealing how fake features can enhance estimation performance even in overparametrized models.

Findings

Fake features can significantly reduce estimation error.

Including more fake features can be beneficial even when overparametrized.

Trade-offs exist between sample size, fake features, and noise assumptions.

Abstract

We consider estimation under model misspecification where there is a model mismatch between the underlying system, which generates the data, and the model used during estimation. We propose a model misspecification framework which enables a joint treatment of the model misspecification types of having fake features as well as incorrect covariance assumptions on the unknowns and the noise. We present a decomposition of the output error into components that relate to different subsets of the model parameters corresponding to underlying, fake and missing features. Here, fake features are features which are included in the model but are not present in the underlying system. Under this framework, we characterize the estimation performance and reveal trade-offs between the number of samples, number of fake features, and the possibly incorrect noise level assumption. In contrast to existing work focusing on incorrect covariance assumptions or missing features, fake features is a central component of our framework. Our results show that fake features can significantly improve the estimation performance, even though they are not correlated with the features in the underlying system. In particular, we show that the estimation error can be decreased by including more fake features in the model, even to the point where the model is overparametrized, i.e., the model contains more unknowns than observations.