On the Success Probability of Three Detectors for the Box-Constrained Integer Linear Model

TL;DR

This paper analyzes and compares the success probabilities of three detectors—maximum likelihood, rounding, and Babai—for a box-constrained integer linear model, providing formulas, bounds, and insights into their relative performances.

Contribution

It derives explicit success probability formulas for the three detectors under different conditions and establishes new bounds and relationships among their performances.

Findings

Success probability formulas for all three detectors are provided.

The rounding detector can outperform Babai and ML detectors in certain cases.

Bounds on the success probability of the ML detector are developed.

Abstract

This paper is concerned with detecting an integer parameter vector inside a box from a linear model that is corrupted with a noise vector following the Gaussian distribution. One of the commonly used detectors is the maximum likelihood detector, which is obtained by solving a box-constrained integer least squares problem, that is NP-hard. Two other popular detectors are the box-constrained rounding and Babai detectors due to their high efficiency of implementation. In this paper, we first present formulas for the success probabilities (the probabilities of correct detection) of these three detectors for two different situations: the integer parameter vector is deterministic and is uniformly distributed over the constraint box. Then, we give two simple examples to respectively show that the success probability of the box-constrained rounding detector can be larger than that of the box-constrained Babai detector and the latter can be larger than the success probability of the maximum likelihood detector when the parameter vector is deterministic, and prove that the success probability of the box-constrained rounding detector is always not larger than that of the box-constrained Babai detector when the parameter vector is uniformly distributed over the constraint box. Some relations between the results for the box constrained and ordinary cases are presented, and two bounds on the success probability of the maximum likelihood detector, which can easily be computed, are developed. Finally, simulation results are provided to illustrate our main theoretical findings.