Adaptative significance levels in linear regression models with known variance

TL;DR

This paper proposes a method to determine adaptive significance levels in Bayesian linear regression models with known variance, optimizing the evidence threshold in the FBST to balance error probabilities as sample size varies.

Contribution

It introduces a novel approach to set adaptive cut-off values for the FBST in linear regression with known variance, accounting for sample size effects.

Findings

Method minimizes combined type I and II errors for given sample sizes.

Adaptive significance levels improve decision accuracy in Bayesian hypothesis testing.

The approach provides a systematic way to choose evidence thresholds based on sample size.

Abstract

The Full Bayesian Significance Test (FBST) for precise hypotheses was presented by Pereira and Stern [Entropy 1(4) (1999) 99-110] as a Bayesian alternative instead of the traditional significance test using p-value. The FBST is based on the evidence in favor of the null hypothesis (H). An important practical issue for the implementation of the FBST is the determination of how large the evidence must be in order to decide for its rejection. In the Classical significance tests, it is known that p-value decreases as sample size increases, so by setting a single significance level, it usually leads H rejection. In the FBST procedure, the evidence in favor of H exhibits the same behavior as the p-value when the sample size increases. This suggests that the cut-off point to define the rejection of H in the FBST should be a sample size function. In this work, the scenario of Linear Regression Models with known variance under the Bayesian approach is considered, and a method to find a cut-off value for the evidence in the FBST is presented by minimizing the linear combination of the averaged type I and type II error probabilities for a given sample size and also for a given dimension of the parametric space.

Tables

Table 1. Table 1: Cut-off values k ∗ superscript 𝑘 k^{*} for e ​ v ​ ( H ; 𝐲 ) 𝑒 𝑣 H 𝐲 ev\left(\text{{H}};\boldsymbol{\mathrm{y}}\right) as function of n 𝑛 n , with d = dim ​ ( 𝚯 ) 𝑑 dim 𝚯 d=\text{dim}(\boldsymbol{\Theta}) , a = b = 1 𝑎 𝑏 1 a=b=1 .

	$k^{\ast }$
$n$	$d=1$	$d=2$
10	0.10040	0.35260
50	0.05166	0.11262
100	0.04447	0.10473
150	0.03776	0.09698
200	0.03179	0.08946
250	0.02667	0.08226
300	0.02244	0.07544
350	0.01905	0.06904
400	0.01639	0.06311
450	0.01429	0.05767
500	0.01264	0.05274
1000	0.00649	0.02823
1500	0.00622	0.02954
2000	0.00610	0.03000

Equations

$$\mathrm{y}=\mathrm{X}\mathbf{\theta }+\mathbf{\epsilon },\mathbf{\epsilon }\sim N_n(0,{\sigma }^2{\mathbb{I}}_n),$$

$$g(\mathbf{\theta })\propto \exp \left\{-\frac{(\mathbf{\theta }-{\mathrm{m}}_0{)}^{\top }{{\mathrm{W}}_0}^{-1}(\mathbf{\theta }-{\mathrm{m}}_0)}{2}\right\}.$$

$${\mathrm{m}}^{\ast }$$

$${\mathrm{W}}^{\ast }$$

$$\boldsymbol{\mathrm{m}_{0}}=\left(\begin{array}[]{c}\boldsymbol{\mathrm{m}_{0}}_{1}\\ \boldsymbol{\mathrm{m}_{0}}_{2}\\ \end{array}\right),~{}~{}~{}~{}~{}~{}\boldsymbol{\mathrm{W}_{0}}=\left(\begin{array}[]{cc}\boldsymbol{\mathrm{W}_{0}}_{11}&\boldsymbol{\mathrm{W}_{0}}_{12}\\ \boldsymbol{\mathrm{W}_{0}}_{21}&\boldsymbol{\mathrm{W}_{0}}_{22}\end{array}\right),$$

$${\mathbf{\theta }}_1\sim N_s\left({{\mathrm{m}}_0}_1,{{\mathrm{W}}_0}_{11}\right),{\mathbf{\theta }}_2\sim N_r\left({{\mathrm{m}}_0}_2,{{\mathrm{W}}_0}_{22}\right),$$

$${\mathbf{\theta }}_1|{\mathbf{\theta }}_2$$

$$:{\mathbf{\theta }}_2=0$$

$$:{\mathbf{\theta }}_2\ne 0$$

$$T_{\mathrm{y}}=\left\{({\mathbf{\theta }}_1,{\mathbf{\theta }}_2)\in \mathbf{\Theta }:f({\mathbf{\theta }}_1,{\mathbf{\theta }}_2|\mathrm{y})>\underset{\text{H}}{\sup }f({\mathbf{\theta }}_1,{\mathbf{\theta }}_2|\mathrm{y})\right\},$$

$$ev\left(\text{H};\mathrm{y}\right)=1-P\left({\chi }_p^2<-2\log \left\{\left[\underset{\text{H}}{\sup }f({\mathbf{\theta }}_1,{\mathbf{\theta }}_2|\mathrm{y})\right]{\left|{\mathrm{W}}^{\ast }\right|}^{1/2}(2\pi {)}^{p/2}\right\}\right),$$

$$\varphi(\boldsymbol{\mathrm{y}})=\left\{\begin{array}[]{l}0\quad if\quad ev\left(\text{{H}};\boldsymbol{\mathrm{y}}\right)>k\\ 1\quad if\quad ev\left(\text{{H}};\boldsymbol{\mathrm{y}}\right)\leq k.\end{array}\right.\;$$

$$\Psi =\left\{\mathrm{y}\in \mathbf{\Omega }:ev\left(\text{H};\mathrm{y}\right)\le k\right\}.$$

$${\alpha }_{\phi }$$

$$={\int }_{\mathrm{y}\in \Psi }{f}_{\text{H}}(\mathrm{y})d\mathrm{y}$$

$$={\int }_{\mathrm{y}\in \Psi }{\int }_{\text{H}}f(\mathrm{y}|{\mathbf{\theta }}_1,{\mathbf{\theta }}_2)g_{\text{H}}({\mathbf{\theta }}_1,{\mathbf{\theta }}_2)d{\mathbf{\theta }}_{1}d{\mathbf{\theta }}_2$$

$$={\int }_{\mathrm{y}\in \Psi }{\int }_{\text{H}}f(\mathrm{y}|{\mathbf{\theta }}_1,{\mathbf{\theta }}_2)g({\mathbf{\theta }}_1|{\mathbf{\theta }}_2=0)d{\mathbf{\theta }}_{1}d{\mathbf{\theta }}_2$$

$$={\int }_{\mathrm{y}\in \Psi }{\int }_{{\mathbf{\theta }}_1\in {\mathbb{R}}^s}f(\mathrm{y}|{\mathbf{\theta }}_1,{\mathbf{\theta }}_2=0)g({\mathbf{\theta }}_1|{\mathbf{\theta }}_2=0)d{\mathbf{\theta }}_1$$

$$={\int }_{\mathrm{y}\in \Psi }{N}_n\left(\mathrm{X}\mathrm{C}{{\mathrm{m}}_0}_{1.2}({\mathbf{\theta }}_2=0),\left({\sigma }^2{\mathbb{I}}_n+(\mathrm{X}\mathrm{C}){{\mathrm{W}}_0}_{11.2}(\mathrm{X}\mathrm{C}{)}^{\top }\right)\right),$$

$${\beta }_{\phi }$$

$$={\int }_{\mathrm{y}\notin \Psi }{f}_{\text{A}}(\mathrm{y})d\mathrm{y}$$

$$={\int }_{\mathrm{y}\notin \Psi }{\int }_{\text{A}}f(\mathrm{y}|\mathbf{\theta })g_{\text{A}}(\mathbf{\theta })d\mathbf{\theta }$$

$$={\int }_{\mathrm{y}\notin \Psi }{\int }_{\text{A}}f(\mathrm{y}|\mathbf{\theta })g(\mathbf{\theta })d\mathbf{\theta }$$

$$={\int }_{\mathrm{y}\notin \Psi }{N}_n\left(\mathrm{X}{\mathrm{m}}_0,\left({\sigma }^2{\mathbb{I}}_n+\mathrm{X}{\mathrm{W}}_0{\mathrm{X}}^{\top }\right)\right).$$

$$\varphi^{*}(\boldsymbol{\mathrm{y}})=\left\{\begin{array}[]{l}0\quad if\quad ev\left(\text{{H}};\boldsymbol{\mathrm{y}}\right)>k^{*}\\ 1\quad if\quad ev\left(\text{{H}};\boldsymbol{\mathrm{y}}\right)\leq k^{*}.\end{array}\right.\;$$

$${\alpha }_{{\phi }^{\ast }}^{\ast }=P({\phi }^{\ast }(\mathrm{y})=1|\text{H}),{\beta }_{{\phi }^{\ast }}^{\ast }=P({\phi }^{\ast }(\mathrm{y})=0|\text{A}).$$

Full text

1 Introduction

The main goal of our work is to determine how small the Bayesian evidence in the FBST should be in order to reject the null hypothesis. Therefore, considering the concepts in Pereira (1985), in Oliveira (2014) and the recent work of Pereira et al. (2017) and Gannon et al. (2019) related to the adaptive significance levels (levels that are function of sample size which are obtained from the generalized form of the Neyman-Pearson Lemma ), we propose to establish a cut-off value $k^{*}$ for the $ev\left(\text{{H}};\boldsymbol{\mathrm{y}}\right)$ as a function of the sample size $n$ and the dimension of the parametric space $d$, i.e., $k^{*}=k^{*}(n,d)$ with $k^{*}\in[0,1]$, such that $k^{*}$ minimizes the linear combination of the averaged type I and type II error probabilities, $a\alpha_{\varphi}+b\beta_{\varphi}$. We will focus on model selection for Linear Regression Models with known variance.

2 Methodology

Consider de normal linear regression model

[TABLE]

where $\boldsymbol{\mathrm{y}}=(y_{1},\dots,y_{n})^{\top}$ is an $n\times 1$ vector of $y_{i}$ observations, $\boldsymbol{\mathrm{X}}=(\boldsymbol{x}_{1},\dots,\boldsymbol{x}_{n})^{\top}$ is an $n\times p$ matrix of known coefficients with $\boldsymbol{x}_{i}=(1,x_{i1},\dots,x_{ip-1})^{\top}$, $\boldsymbol{\theta}=(\boldsymbol{\theta}_{1}^{\top},\boldsymbol{\theta}_{2}^{\top})^{\top}$ is a $p\times 1$ vector of parameters, and $\boldsymbol{\varepsilon}=(\varepsilon_{1},\dots,\varepsilon_{n})^{\top}$ an $n\times 1$ vector of random errors. Suppose that the residual error variance $\sigma^{2}$ is known, then $f(\boldsymbol{\mathrm{y}}|\boldsymbol{\theta})\sim N_{n}(\boldsymbol{\mathrm{X}}\boldsymbol{\theta},\sigma^{2}\mathbb{I}_{n})$. The natural conjugate prior family is the family of normal distributions. Suppose therefore that $\boldsymbol{\theta}$ has the $N_{p}(\boldsymbol{\mathrm{m}_{0}},\boldsymbol{\mathrm{W}_{0}})$ prior distribution

[TABLE]

Then, the posterior distribution of $\boldsymbol{\theta}$ is $\boldsymbol{\theta}|\boldsymbol{\mathrm{y}}\sim N_{p}(\boldsymbol{\mathrm{m}^{*}},\boldsymbol{\mathrm{W}^{*}})$, with

[TABLE]

If $\boldsymbol{\theta}_{1}$ has $s$ elements and $\boldsymbol{\theta}_{2}$ has $r$ elements write

[TABLE]

where $\boldsymbol{\mathrm{m}_{0}}_{1}$ is $s\times 1$, $\boldsymbol{\mathrm{W}_{0}}_{11}$ is $s\times s$, $\boldsymbol{\mathrm{m}_{0}}_{2}$ is $r\times 1$, $\boldsymbol{\mathrm{W}_{0}}_{22}$ is $r\times r$. So,

[TABLE]

Using general results on multivariate normal distributions,

[TABLE]

where $\boldsymbol{\mathrm{m}_{0}}_{1.2}(\boldsymbol{\theta}_{2})=\boldsymbol{\mathrm{m}_{0}}_{1}+\boldsymbol{\mathrm{W}_{0}}_{12}\boldsymbol{\mathrm{W}_{0}}_{22}^{-1}(\boldsymbol{\theta}_{2}-\boldsymbol{\mathrm{m}_{0}}_{2})$ and $\boldsymbol{\mathrm{W}_{0}}_{11.2}=\boldsymbol{\mathrm{W}_{0}}_{11}-\boldsymbol{\mathrm{W}_{0}}_{12}\boldsymbol{\mathrm{W}_{0}}_{22}^{-1}\boldsymbol{\mathrm{W}_{0}}_{21}$. A corresponding distribution result if we change $\boldsymbol{\mathrm{m}_{0}}$ to $\boldsymbol{\mathrm{m}^{*}}$ and $\boldsymbol{\mathrm{W}_{0}}$ to $\boldsymbol{\mathrm{W}^{*}}$.

Definition 1**.**

Let $f(\boldsymbol{\theta}|\boldsymbol{\mathrm{y}})$ be the posterior density of $\boldsymbol{\theta}$ given the observed sample. Consider a sharp hypothesis $\text{{H}}:\boldsymbol{\theta}\in\boldsymbol{\Theta}_{\text{{H}}}$ and let ${T_{\boldsymbol{\mathrm{y}}}=\left\{\boldsymbol{\theta}\in\boldsymbol{\Theta}:f(\boldsymbol{\theta}|\boldsymbol{\mathrm{y}})>\text{sup}_{\text{{H}}}f(\boldsymbol{\theta}|\boldsymbol{\mathrm{y}})\right\}}$ be the set tangential to H. The measure of evidence in favor H is defined as ${ev\left(\text{{H}};\boldsymbol{\mathrm{y}}\right)=1-P(\boldsymbol{\theta}\in T_{\boldsymbol{\mathrm{y}}}|\boldsymbol{\mathrm{y}})}$. The FBST is the procedure that rejects H whenever $ev\left(\text{{H}};\boldsymbol{\mathrm{y}}\right)$ is small (Pereira et al., 2008).

Suppose that we want to test the hypotheses

[TABLE]

The tangential set to the null hypothesis is

[TABLE]

and, since $(\boldsymbol{\theta}-\boldsymbol{\mathrm{m}^{*}})^{\top}\boldsymbol{\mathrm{W}^{*}}^{-1}(\boldsymbol{\theta}-\boldsymbol{\mathrm{m}^{*}})\sim\chi^{2}_{p}$, the evidence in favor of H is

[TABLE]

where, $\underset{\text{{H}}}{\operatorname{sup}}f(\boldsymbol{\theta}_{1},\boldsymbol{\theta}_{2}|\boldsymbol{\mathrm{y}})=f(\boldsymbol{\mathrm{m}^{*}}_{1.2}(\boldsymbol{\theta}_{2}=\boldsymbol{0}),\,\boldsymbol{0}|\boldsymbol{\mathrm{y}})$.

Consider $\varphi(\boldsymbol{\mathrm{y}})$ as the test such that

[TABLE]

Thus, define the set

[TABLE]

The averaged error probabilities can be expressed in terms of the Bayesian prior predictive densities under the respective hypotheses as follows

[TABLE]

where $\boldsymbol{\mathrm{C}}_{(s+r)\times s}=[\mathbb{I}_{s},\boldsymbol{0}_{s\times r}]^{\top}$.

[TABLE]

So, the adaptive cut-off value $k^{*}$ for $ev\left(\text{{H}};x\right)$ will be the $k$ that minimizes $a\alpha_{\varphi}+b\beta_{\varphi}$.

Finally, define $\varphi^{*}(\boldsymbol{\mathrm{y}})$ as the test such that

[TABLE]

The optimal averaged error probabilities that depend on the sample size will be

[TABLE]

3 Results

By increasing $n$, $k^{*}$ shows a decreasing trend, which means that the influence of sample size on the determination of the cut-off for $ev\left(\text{{H}};\boldsymbol{\mathrm{y}}\right)$ is very relevant.

On the other hand, it is possible to notice the differences in the results between the two models. Then, the cut-off value for $ev\left(\text{{H}};\boldsymbol{\mathrm{y}}\right)$ will depend not only on the sample size but also on the dimension of the parametric space. More specifically, the $k^{*}$ value is greater when $d$ is higher.

With this procedure, increasing the sample size implies that the probabilities of both kind of errors and their linear combination decrease, when in most cases, setting a single level of significance independent of sample size, only type II error probability decreases.