A Rigorous Analysis of Least Squares Sine Fitting Using Quantized Data: the Random Phase Case

TL;DR

This paper rigorously analyzes least squares sine fitting with quantized data, revealing estimator bias, inconsistency, and variance under realistic conditions including offsets and noise.

Contribution

It provides a comprehensive, model-based analysis of the estimator's properties without simplifying quantization assumptions, highlighting its limitations.

Findings

Estimator is inconsistent and biased.

Variance may be underestimated with simple models.

Effects of offsets and noise are incorporated into the analysis.

Abstract

This paper considers least-square based estimation of the amplitude and square amplitude of a quantized sine wave, done by considering random initial record phase. Using amplitude- and frequency-domain modeling techniques, it is shown that the estimator is inconsistent, biased and has a variance that may be underestimated if the simple model of quantization is applied. The effects of both sine wave offset values and additive Gaussian noise are taken into account. General estimator properties are derived, without making simplifying assumptions on the role of the quantization process, to allow assessment of measurement uncertainty, when this least-square procedure is used.