Approximation Properties of Variational Bayes for Vector Autoregressions

TL;DR

This paper analyzes the approximation accuracy of Variational Bayes in vector autoregressions, showing it accurately estimates the posterior mean and predictive densities, with insights into variance underestimation.

Contribution

It derives the approximation error of VB for linear Gaussian regressions, revealing conditions for accurate posterior mean and predictive densities.

Findings

VB approximates the posterior mean perfectly.

VB underestimates posterior variance and mode.

VB estimates predictive densities accurately.

Abstract

Variational Bayes (VB) is a recent approximate method for Bayesian inference. It has the merit of being a fast and scalable alternative to Markov Chain Monte Carlo (MCMC) but its approximation error is often unknown. In this paper, we derive the approximation error of VB in terms of mean, mode, variance, predictive density and KL divergence for the linear Gaussian multi-equation regression. Our results indicate that VB approximates the posterior mean perfectly. Factors affecting the magnitude of underestimation in posterior variance and mode are revealed. Importantly, We demonstrate that VB estimates predictive densities accurately.

Tables

Table 1. Table 1: Comparing various features of the exact and MFVB posteriors for natural conjugate VARs

	Exact Posterior	MFVB Posterior
Marginal Posterior	$\begin{array}{c}p\left(𝚪|𝐘\right)=ℳ𝒯(\overline{𝚪},\overline{𝐒},\overline{𝐕},\overline{\upsilon })\\ p\left({𝚺}^{-1}|𝐘\right)=𝒲({\overline{𝐒}}^{-1},\overline{\upsilon })\overline{\upsilon }=T+\underset{¯}{\upsilon }\end{array}$	$\begin{array}{c}q\left(𝚪\right)=ℳ𝒩\{\overline{𝚪},{\overline{{𝚺}^{-1}}}^{-1},\overline{𝐕}\}\\ q\left({𝚺}^{-1}\right)=𝒲(\overline{𝐒}_q^{}{}_{}{}^{-1},{\overline{\upsilon }}_q){\overline{\upsilon }}_q=T+\underset{¯}{\upsilon }+p\end{array}$
Posterior Mean	$\begin{array}{c}{E}_p\left(𝚪\right)=\overline{𝚪}\\ E_p\left({𝚺}^{-1}\right)=\overline{\upsilon }{\overline{𝐒}}^{-1}\end{array}$	$\begin{array}{c}{E}_q\left(𝚪\right)=\overline{𝚪}\\ E_q\left({𝚺}^{-1}\right)={\overline{\upsilon }}_q\overline{𝐒}_q^{}{}_{}{}^{-1}=\overline{\upsilon }{\overline{𝐒}}^{-1}\end{array}$
Posterior Mode	$\begin{array}{c}Mod{e}_p\left(𝚪\right)=\overline{𝚪}\\ Mod{e}_p\left({𝚺}^{-1}\right)=\left(\overline{\upsilon }+p-m-1\right){\overline{𝐒}}^{-1}\end{array}$	$\begin{array}{c}Mod{e}_q\left(𝚪\right)=\overline{𝚪}\\ Mod{e}_q\left({𝚺}^{-1}\right)=\frac{\overline{\upsilon }}{{\overline{\upsilon }}_q}\left(\overline{\upsilon }+p-m-1\right){\overline{𝐒}}^{-1}\end{array}$
Posterior Variance	$\begin{array}{c}Va{r}_p\left(\gamma \right)=\frac{1}{\overline{\upsilon }-M-1}\overline{𝐒}\otimes \overline{𝐕}\\ Va{r}_p\left({\sigma }_{ij}^{\ast }\right)=\overline{\upsilon }(s{}_{ij}{}^{\ast }{}^2+s_{ii}^{\ast }{s}_{jj}^{\ast })\\ \text{where}{𝚺}^{-1}=\left\{{\sigma }_{ij}^{\ast }\right\}{\overline{𝐒}}^{-1}=\left\{s_{ij}^{\ast }\right\}\end{array}$	$\begin{array}{c}Va{r}_q\left(\gamma \right)=\frac{1}{\overline{\upsilon }}\overline{𝐒}\otimes \overline{𝐕}\\ Va{r}_q\left({\sigma }_{ij}^{\ast }\right)=\frac{{\overline{\upsilon }}^2}{{\overline{\upsilon }}_q}(s{}_{ij}{}^{\ast }{}^2+s_{ii}^{\ast }{s}_{jj}^{\ast })\\ \text{where}{𝚺}^{-1}=\left\{{\sigma }_{ij}^{\ast }\right\}{\overline{𝐒}}^{-1}=\left\{s_{ij}^{\ast }\right\}\end{array}$
Log of Marginal Likelihood	$\begin{array}{c}-\frac{MT}{2}\ln \pi +\frac{M}{2}\left(\ln \left|\overline{𝐕}\right|-\ln \left|\underset{¯}{𝐕}\right|\right)-\frac{\overline{\upsilon }}{2}\ln \left|\overline{𝐒}\right|\\ +\frac{\underset{¯}{\upsilon }}{2}\ln \left|\underset{¯}{𝐒}\right|+\ln {\Gamma }_M\left(\frac{\overline{\upsilon }}{2}\right)-\ln {\Gamma }_M\left(\frac{\underset{¯}{\upsilon }}{2}\right)\end{array}$	$\begin{array}{c}-\frac{MT}{2}\ln \pi +\frac{M}{2}\left(\ln \left|\overline{𝐕}\right|-\ln \left|\underset{¯}{𝐕}\right|\right)+\frac{\overline{\upsilon }}{2}\ln \left|{\overline{𝐒}}^{-1}\right|+\frac{\underset{¯}{\upsilon }}{2}\ln \left|\underset{¯}{𝐒}\right|\\ -\frac{Mp}{2}\ln 2e+\frac{M}{2}\ln \left(\frac{{\overline{\upsilon }}^{\overline{\upsilon }}}{\overline{\upsilon }_q^{}{}_{}{}^{{\overline{\upsilon }}_q}}\right)+\ln {\Gamma }_M\left(\frac{{\overline{\upsilon }}_q}{2}\right)-\ln {\Gamma }_M\left(\frac{\underset{¯}{\upsilon }}{2}\right)\end{array}$
Predictive Density	$\begin{array}{c}\text{t-distribution with}\\ E_p\left({𝐲}_{T+1}|𝐘\right)={\left({𝐱}_{T+1}\overline{𝚪}\right)}^{\prime }\\ Va{r}_p\left({𝐲}_{T+1}|𝐘\right)=\left(1+{𝐱}_{T+1}\overline{𝐕}𝐱_{}^{\prime }{}_{T+1}{}^{}\right)\frac{\overline{𝐒}}{\overline{\upsilon }-2}\end{array}$	$\begin{array}{c}\text{Sum of a normal and a t-distribution with}\\ E_q\left({𝐲}_{T+1}|𝐘\right)={\left({𝐱}_{T+1}\overline{𝚪}\right)}^{\prime }\\ Va{r}_q\left({𝐲}_{T+1}|𝐘\right)=\left(\frac{{\overline{\upsilon }}_q}{{\overline{\upsilon }}_q-2}+{𝐱}_{T+1}\overline{𝐕}𝐱_{}^{\prime }{}_{T+1}{}^{}\right)\frac{\overline{𝐒}}{\overline{\upsilon }}\end{array}$

Table 2. Table 2: Comparisons for conjugate VARs of different sizes

VAR	$\frac{Va{r}_q\left(𝚪\right)}{Va{r}_p\left(𝚪\right)}$	$\frac{Va{r}_q\left({\sigma }_{ij}^{\ast }\right)}{Va{r}_p\left({\sigma }_{ij}^{\ast }\right)}$	$\frac{Mod{e}_q\left({\sigma }_{ij}^{\ast }\right)}{Mod{e}_p\left({\sigma }_{ij}^{\ast }\right)}$	$\frac{Va{r}_q\left({𝐲}_{T+1}|𝐘\right)}{Va{r}_p\left({𝐲}_{T+1}|𝐘\right)}$
Small M=3	0.980	0.930	0.939	0.990
Medium M=7	0.961	0.853	0.876	0.990
Large M=20	0.903	0.622	0.728	0.991

Table 3. Table 3: KL for conjugate and independent VARs

Model	Size	$\ln ML$	$\ln \underset{¯}{ML}$	KL
Conjugate VAR	Small	1370.62	1370.43	0.189
Conjugate VAR	Medium	2910.09	2908.22	1.874
Independent VAR	Small	1368.80	1368.61	0.193
Independent VAR	Medium	2882.12	2880.38	1.742

Table 4. Table 4: Comparing posterior mean and   variance of some parameters for small VAR

Conjugate VAR		Independent VAR
$\frac{E_q\left({\sigma }_{ii}^{\ast }\right)}{E_p\left({\sigma }_{ii}^{\ast }\right)}$	$\frac{Va{r}_q\left({\sigma }_{ii}^{\ast }\right)}{Va{r}_p\left({\sigma }_{ii}^{\ast }\right)}$	$\frac{E_q\left({\sigma }_{ii}^{\ast }\right)}{E_p\left({\sigma }_{ii}^{\ast }\right)}$	$\frac{Va{r}_q\left({\sigma }_{ii}^{\ast }\right)}{Va{r}_p\left({\sigma }_{ii}^{\ast }\right)}$
1	0.939	1.001	0.941
1	0.939	1.002	0.947
1	0.939	0.999	0.927

Table 5. Table 5: Predictive density’s mean and variance

Conjugate VAR		Independent VAR
$\frac{E_q\left({𝐲}_{T+1}|𝐘\right)}{E_p\left({𝐲}_{T+1}|𝐘\right)}$	$\frac{Va{r}_q\left({𝐲}_{T+1}|𝐘\right)}{Va{r}_p\left({𝐲}_{T+1}|𝐘\right)}$	$\frac{E_q\left({𝐲}_{T+1}|𝐲\right)}{E_p\left({𝐲}_{T+1}|𝐲\right)}$	$\frac{Va{r}_q\left({𝐲}_{T+1}|𝐲\right)}{Va{r}_p\left({𝐲}_{T+1}|𝐲\right)}$
1	0.9987	1.0000	0.9923
1	0.9987	1.0000	0.9874
1	0.9987	0.9980	0.9869

Table 6. Table 6: Mean and variance ratios of MFVB to MCMC for medium VAR (M=7) with the independent prior

Variances		Predictive Density
$\frac{E_q\left({\sigma }_{ii}^{\ast }\right)}{E_p\left({\sigma }_{ii}^{\ast }\right)}$	$\frac{Va{r}_q\left({\sigma }_{ii}^{\ast }\right)}{Va{r}_p\left({\sigma }_{ii}^{\ast }\right)}$	$\frac{E_q\left({𝐲}_{T+1}|𝐲\right)}{E_p\left({𝐲}_{T+1}|𝐲\right)}$	$\frac{Va{r}_q\left({𝐲}_{T+1}|𝐲\right)}{Va{r}_p\left({𝐲}_{T+1}|𝐲\right)}$
0.9993	0.8809	1.0000	0.9936
0.9984	0.8811	1.0000	0.9919
0.9984	0.8776	1.0173	0.9930
0.9991	0.8819	1.0000	0.9911
0.9989	0.8710	0.9998	0.9827
0.9993	0.9029	1.0000	0.9966
0.9987	0.9217	1.0000	0.9887

Equations

$$KL(q||p)={\int }_{\theta \in \Theta }q(\theta )\ln [q(\theta )/p(\theta |\mathbf{y})]d\theta$$

$$\begin{array}{rl}& q_1{({\theta }_1)}^{new}\leftarrow \frac{\exp (E_{{\theta }_{-1}}\log [p(\mathbf{y},\theta )])}{\int \exp (E_{{\theta }_{-1}}\log [p(\mathbf{y},\theta )])d{\theta }_1}\\ & ⋮⋮\\ & ⋮⋮\\ & q_M{({\theta }_M)}^{new}\leftarrow \frac{\exp (E_{{\theta }_{-M}}\log [p(\mathbf{y},\theta )])}{\int \exp (E_{{\theta }_{-M}}\log [p(\mathbf{y},\theta )])d{\theta }_M}\end{array}$$

$$\ln ML=E_q\ln p(\mathbf{y},\theta )-E_q\ln q(\theta )+KL(q||p)$$

$$\ln \underline{ML}=E_q\ln p(\mathbf{y},\theta )-E_q\ln q(\theta )$$

$${\mathbf{y}}_t={\gamma }_0+{\mathbf{\Gamma }}_1{\mathbf{y}}_{t-1}+....+{\mathbf{\Gamma }}_d{\mathbf{y}}_{t-d}+{\epsilon }_t$$

$$\mathbf{Y}=\mathbf{X\Gamma }+\mathbf{E}\text{with}\mathbf{E}\sim \mathcal{M}\mathcal{N}(0,\mathbf{\Sigma },{\mathbf{I}}_T)$$

$$\begin{array}{rl}& p(\mathbf{\Gamma },{\mathbf{\Sigma }}^{-1})=p(\mathbf{\Gamma }|\mathbf{\Sigma })p({\mathbf{\Sigma }}^{-1})\\ & p(\mathbf{\Gamma }|\mathbf{\Sigma })=\mathcal{M}\mathcal{N}(\underline{\mathbf{\Gamma }},\mathbf{\Sigma },\underline{\mathbf{V}})\&p({\mathbf{\Sigma }}^{-1})=\mathcal{W}({\underline{\mathbf{S}}}^{-1},\underline{\upsilon })\end{array}$$

$$\begin{array}{rl}& p(\mathbf{\Gamma }|{\mathbf{\Sigma }}^{-1},\mathbf{Y})=\mathcal{M}\mathcal{N}(\bar{\mathbf{\Gamma }},\mathbf{\Sigma },\bar{\mathbf{V}})\\ & \bar{\mathbf{V}}={[{\underline{\mathbf{V}}}^{-1}+{\mathbf{X}}^{\prime }\mathbf{X}]}^{-1}\&\bar{\mathbf{\Gamma }}=\bar{\mathbf{V}}[{\underline{\mathbf{V}}}^{-1}\underline{\mathbf{\Gamma }}+{\mathbf{X}}^{\prime }\mathbf{Y}]\\ & p({\mathbf{\Sigma }}^{-1}|\mathbf{Y})=\mathcal{W}({\bar{\mathbf{S}}}^{-1},\bar{\upsilon })\text{where}\bar{\upsilon }=T+\underline{\upsilon }\\ & \bar{\mathbf{S}}=(\mathbf{Y}-\mathbf{X}\bar{\mathbf{\Gamma }}{)}^{\prime }(\mathbf{Y}-\mathbf{X}\bar{\mathbf{\Gamma }})+\underline{\mathbf{S}}+(\bar{\mathbf{\Gamma }}-\underline{\mathbf{\Gamma }}{)}^{\prime }{\underline{\mathbf{V}}}^{-1}(\bar{\mathbf{\Gamma }}-\underline{\mathbf{\Gamma }})\end{array}$$

$$\begin{array}{rl}& q(\mathbf{\Gamma })=\mathcal{M}\mathcal{N}\{\bar{\mathbf{\Gamma }},{\overline{{\mathbf{\Sigma }}^{-1}}}^{-1},\bar{\mathbf{V}}\}\\ & \bar{\mathbf{V}}={[{\underline{\mathbf{V}}}^{-1}+{\mathbf{X}}^{\prime }\mathbf{X}]}^{-1}\bar{\mathbf{\Gamma }}=\bar{\mathbf{V}}[{\underline{\mathbf{V}}}^{-1}\underline{\mathbf{\Gamma }}+{\mathbf{X}}^{\prime }\mathbf{Y}]\\ & \overline{{\mathbf{\Sigma }}^{-1}}=\bar{\upsilon }{\bar{\mathbf{S}}}^{-1}\\ & \bar{\mathbf{S}}=(\mathbf{Y}-\mathbf{X}\bar{\mathbf{\Gamma }}{)}^{\prime }(\mathbf{Y}-\mathbf{X}\bar{\mathbf{\Gamma }})+\underline{\mathbf{S}}+(\bar{\mathbf{\Gamma }}-\underline{\mathbf{\Gamma }}{)}^{\prime }{\underline{\mathbf{V}}}^{-1}(\bar{\mathbf{\Gamma }}-\underline{\mathbf{\Gamma }})\\ & q({\mathbf{\Sigma }}^{-1})=\mathcal{W}({{\bar{\mathbf{S}}}_q}^{-1},{\bar{\upsilon }}_q)\\ & {\bar{\mathbf{S}}}_q=\frac{{\bar{\upsilon }}_q}{\bar{\upsilon }}\bar{\mathbf{S}}{\bar{\upsilon }}_q=T+p+\underline{\upsilon }\end{array}$$

$$\left(\begin{array}{rl}& {\mathbf{y}}_1\\ & ⋮\\ & {\mathbf{y}}_T\end{array}\right)=\left(\begin{array}{rl}& {\mathbf{Z}}_1\\ & ⋮\\ & {\mathbf{Z}}_T\end{array}\right)\left(\begin{array}{rl}& {\beta }_1\\ & ⋮\\ & {\beta }_M\end{array}\right)+\left(\begin{array}{rl}& {\epsilon }_1\\ & ⋮\\ & {\epsilon }_M\end{array}\right)$$

$$\mathbf{y}=\mathbf{Z}\beta +\epsilon \text{with}\epsilon \sim \mathcal{N}(0,{\mathbf{I}}_T\otimes {\mathbf{\Sigma }}_M)$$

$$\begin{array}{rl}& p(\beta ,{\mathbf{\Sigma }}^{-1})=p(\beta )p({\mathbf{\Sigma }}^{-1})\\ & \text{ with}p(\beta )=\mathcal{N}(\underline{\beta },\underline{\mathbf{V}}),p({\mathbf{\Sigma }}^{-1})=\mathcal{W}({\underline{\mathbf{S}}}^{-1},\underline{\upsilon })\end{array}$$

$$\begin{array}{rl}& p(\beta |{\mathbf{\Sigma }}^{-1},\mathbf{y})=\mathcal{N}(\bar{\beta },\bar{\mathbf{V}})\\ & \bar{\mathbf{V}}\text{=}[{\underline{\mathbf{V}}}^{-1}+\sum _{t=1}^T{{\mathbf{Z}}^{\prime }}_t{\mathbf{\Sigma }}^{-1}{\mathbf{Z}}_t{]}^{-1},\bar{\beta }=\bar{\mathbf{V}}[{\underline{\mathbf{V}}}^{-1}\underline{\beta }+\sum _{t=1}^T{{\mathbf{Z}}^{\prime }}_t{\mathbf{\Sigma }}^{-1}{\mathbf{y}}_t]\\ & p({\mathbf{\Sigma }}^{-1}|\beta ,\mathbf{y})=\mathcal{W}({\bar{\mathbf{S}}}^{-1},\bar{\upsilon })\bar{\upsilon }=T+\underline{\upsilon }\\ & \bar{\mathbf{S}}\text{=}\underline{\mathbf{S}}+\sum _{t=1}^T({\mathbf{y}}_t-{\mathbf{Z}}_t\beta )({\mathbf{y}}_t-{\mathbf{Z}}_t\beta {)}^{\prime }\end{array}$$

$$\begin{array}{rl}& q(\beta )=\mathcal{N}({\bar{\beta }}_q,{\bar{\mathbf{V}}}_q)\\ & q({\mathbf{\Sigma }}^{-1})=\mathcal{W}({\bar{\mathbf{S}}}_q^{-1},\bar{\upsilon })\bar{\upsilon }=T+\underline{\upsilon }\end{array}$$

$$\begin{array}{rl}& {\bar{\mathbf{V}}}_q\text{=}[{\underline{\mathbf{V}}}^{-1}+\sum _{t=1}^T{{\mathbf{Z}}^{\prime }}_t\overline{{\mathbf{\Sigma }}_q^{-1}}{\mathbf{Z}}_t{]}^{-1},{\bar{\beta }}_q=\bar{\mathbf{V}}[{\underline{\mathbf{V}}}^{-1}\underline{\beta }+\sum _{t=1}^T{{\mathbf{Z}}^{\prime }}_t\overline{{\mathbf{\Sigma }}_q^{-1}}{\mathbf{y}}_t]\\ & {\bar{\mathbf{S}}}_q\text{=}\underline{\mathbf{S}}+\sum _{t=1}^T({\mathbf{y}}_t-{\mathbf{Z}}_t{\bar{\beta }}_q)({\mathbf{y}}_t-{\mathbf{Z}}_t{\bar{\beta }}_q{)}^{\prime }+{\mathbf{Z}}_t{\bar{\mathbf{V}}}_q{{\mathbf{Z}}^{\prime }}_t,\overline{{\mathbf{\Sigma }}_q^{-1}}=\bar{\upsilon }{\bar{\mathbf{S}}}_q^{-1}\end{array}$$

$$\begin{array}{rl}& \ln \underline{ML}=\frac{p}{2}-\frac{MT}{2}\ln \pi +\ln {\Gamma }_M\left(\frac{\bar{\upsilon }}{2}\right)-\ln {\Gamma }_M\left(\frac{\underline{\upsilon }}{2}\right)+\frac{\ln \left|{\bar{\mathbf{V}}}_q\right|-\ln |\underline{\mathbf{V}}|}{2}\\ & +\frac{\bar{\upsilon }\ln \left|{\bar{\mathbf{S}}}_q^{-1}\right|-\underline{\upsilon }\ln \left|{\underline{\mathbf{S}}}^{-1}\right|}{2}-\frac{tr\left\{{\underline{\mathbf{V}}}^{-1}[({\bar{\beta }}_q-\underline{\beta })({\bar{\beta }}_q-\underline{\beta }{)}^{\prime }+{\bar{\mathbf{V}}}_q]\right\}}{2}\end{array}$$

$$\begin{array}{rl}& E_q({\mathbf{y}}_{T+1}|\mathbf{y})=E_q[{\mathbf{Z}}_{T+1}\beta |\mathbf{y}]+E_q({\epsilon }_{T+1}|\mathbf{y})={\mathbf{Z}}_{T+1}{\bar{\beta }}_q\\ & Va{r}_q({\mathbf{y}}_{T+1}|\mathbf{Y})=Va{r}_q[{\mathbf{Z}}_{T+1}{\beta }_q|\mathbf{y}]+Va{r}_q({\epsilon }_{T+1}|\mathbf{y})\\ & ={\mathbf{Z}}_{T+1}{\bar{\mathbf{V}}}_q{{\mathbf{Z}}^{\prime }}_{T+1}+\frac{{\bar{\mathbf{S}}}_q}{(\bar{\upsilon }-2)}\end{array}$$

$$KL=\ln ML-\ln \underline{ML}=-\frac{Mp}{2}(\ln 2+1)+\frac{M}{2}\ln \left(\frac{{{\bar{\upsilon }}_q}^{{\bar{\upsilon }}_q}}{{\bar{\upsilon }}^{\bar{\upsilon }}}\right)-\ln \left(\frac{{\Gamma }_M({\bar{\upsilon }}_q/2)}{{\Gamma }_M\left(\bar{\upsilon }/2\right)}\right)$$

$$KL\sim \frac{1}{2}\sum _{j=1}^M\ln \left(\frac{{\left(\bar{\upsilon }-j+1\right)}^{(\bar{\upsilon }-j)}{{\bar{\upsilon }}_q}^{{\bar{\upsilon }}_q}}{{\left({\bar{\upsilon }}_q-j+1\right)}^{({\bar{\upsilon }}_q-j)}{\bar{\upsilon }}^{\bar{\upsilon }}}\right)$$

$$T\to \infty \Rightarrow \frac{Va{r}_q({\sigma }_{ij}^{\ast })}{Va{r}_p({\sigma }_{ij}^{\ast })}\to 1,\frac{Va{r}_q({\gamma }_h)}{Va{r}_p({\gamma }_h)}\to 1,KL\to 0$$

$$\underline{\upsilon }\to \infty \Rightarrow \frac{Va{r}_q({\sigma }_{ij}^{\ast })}{Va{r}_p({\sigma }_{ij}^{\ast })}\to 1,\frac{Va{r}_q({\gamma }_h)}{Va{r}_p({\gamma }_h)}\to 1,KL\to 0$$

$$p\to \infty \Rightarrow \frac{Va{r}_q({\sigma }_{ij}^{\ast })}{Va{r}_p({\sigma }_{ij}^{\ast })}\to 0,KL\to \infty$$

$$\begin{array}{rl}& \ln p(\mathbf{Y},\mathbf{\Gamma },{\mathbf{\Sigma }}^{-1})=C+\frac{1}{2}(T+p+\underline{\upsilon }-M-1)\ln \left|{\mathbf{\Sigma }}^{-1}\right|\\ & -\frac{1}{2}tr\{{\mathbf{\Sigma }}^{-1}[(\mathbf{Y}-\mathbf{X\Gamma }{)}^{\prime }(\mathbf{Y}-\mathbf{X\Gamma })+(\mathbf{\Gamma }-\underline{\mathbf{\Gamma }}{)}^{\prime }{\underline{\mathbf{V}}}^{-1}(\mathbf{\Gamma }-\underline{\mathbf{\Gamma }})+\underline{\mathbf{S}}]\}\end{array}$$

$$\begin{array}{rl}& q(\mathbf{\Gamma })\leftarrow \mathcal{M}\mathcal{N}\{\bar{\mathbf{\Gamma }},{\overline{{\mathbf{\Sigma }}^{-1}}}^{-1},\bar{\mathbf{V}}\}\\ & \bar{\mathbf{V}}={[{\underline{\mathbf{V}}}^{-1}+{\mathbf{X}}^{\prime }\mathbf{X}]}^{-1}\bar{\mathbf{\Gamma }}=\bar{\mathbf{V}}[{\underline{\mathbf{V}}}^{-1}\underline{\mathbf{\Gamma }}+{\mathbf{X}}^{\prime }\mathbf{Y}]\end{array}$$

$$q({\mathbf{\Sigma }}^{-1})\leftarrow \mathcal{W}({{\bar{\mathbf{S}}}_q}^{-1},{\bar{\upsilon }}_q){\bar{\upsilon }}_q=T+p+\underline{\upsilon }$$

$${\bar{\mathbf{S}}}_q=(\mathbf{Y}-\mathbf{X}\bar{\mathbf{\Gamma }}{)}^{\prime }(\mathbf{Y}-\mathbf{X}\bar{\mathbf{\Gamma }})+\underline{\mathbf{S}}+(\bar{\mathbf{\Gamma }}-\underline{\mathbf{\Gamma }}{)}^{\prime }{\underline{\mathbf{V}}}^{-1}(\bar{\mathbf{\Gamma }}-\underline{\mathbf{\Gamma }})+\mathbf{\Omega }$$

$$\mathrm{cov}({\mathbf{\Gamma }}_i,{\mathbf{\Gamma }}_j){=}^{}{\bar{\sigma }}_{ij}^{+}\bar{\mathbf{V}}$$

$$\mathbf{\Omega }={\overline{{\mathbf{\Sigma }}^{-1}}}^{-1}=\frac{p}{T+p+\underline{\upsilon }}{\bar{\mathbf{S}}}_q$$

$${\bar{\mathbf{S}}}_q=(\mathbf{Y}-\mathbf{X}\bar{\mathbf{\Gamma }}{)}^{\prime }(\mathbf{Y}-\mathbf{X}\bar{\mathbf{\Gamma }})+\underline{\mathbf{S}}+(\bar{\mathbf{\Gamma }}-\underline{\mathbf{\Gamma }}{)}^{\prime }{\underline{\mathbf{V}}}^{-1}(\bar{\mathbf{\Gamma }}-\underline{\mathbf{\Gamma }})+\frac{p}{T+p+\underline{\upsilon }}{\bar{\mathbf{S}}}_q$$

$$\displaystyle\quad\ \ =\frac{{{{\bar{\upsilon}}}_{q}}}{{\bar{\upsilon}}}[(\mathbf{Y}-\mathbf{X\bar{\Gamma}}{)}^{\prime}(\mathbf{Y}-\mathbf{X\bar{\Gamma}})\mathbf{+\underline{S}+}(\mathbf{\bar{\Gamma}}-\mathbf{\underline{\Gamma}}{)}^{\prime}{{{\mathbf{\underline{V}}}}^{-1}}(\mathbf{\bar{\Gamma}}-\mathbf{\underline{\Gamma}})]\$$

$$\begin{array}{rl}& q({\mathbf{x}}_{T+1}\mathbf{\Gamma }|\mathbf{Y})=\mathcal{M}\mathcal{N}\left({\mathbf{x}}_{T+1}\bar{\mathbf{\Gamma }},{\overline{{\mathbf{\Sigma }}^{-1}}}^{-1},{\mathbf{x}}_{T+1}{\bar{\mathbf{V}}{\mathbf{x}}^{\prime }}_{T+1}\right)\\ & =\mathcal{N}\left({\mathbf{x}}_{T+1}\bar{\mathbf{\Gamma }},{\overline{{\mathbf{\Sigma }}^{-1}}}^{-1}\otimes {\mathbf{x}}_{T+1}{\bar{\mathbf{V}}{\mathbf{x}}^{\prime }}_{T+1}\right)\\ & =\mathcal{N}\left({\mathbf{x}}_{T+1}\bar{\mathbf{\Gamma }},\frac{\bar{\mathbf{S}}}{\bar{\upsilon }}{\mathbf{x}}_{T+1}{\bar{\mathbf{V}}{\mathbf{x}}^{\prime }}_{T+1}\right)\end{array}$$

$$q({\epsilon }_{T+1}|\mathbf{Y})=\int p({\epsilon }_{T+1}|{\mathbf{\Sigma }}^{-1},\mathbf{Y})q({\mathbf{\Sigma }}^{-1}|\mathbf{Y})d{\mathbf{\Sigma }}^{-1}=\mathcal{T}(0,\frac{1}{{\bar{\upsilon }}_q}{\bar{\mathbf{S}}}_q,{\bar{\upsilon }}_q)$$

Full text

Mean-Field, VAR, Big Data, Machine Learning, Marginal Likelihood,

LaTeX 2ε,

\startlocaldefs\endlocaldefs

T1I Would like to thank Bill Griffiths and Tomasz Woźniak for their very useful comments on the paper

t1Part of this research was done while the author was a visiting fellow at the Centre for Transformative Innovation

1 Introduction

Variational Bayes (VB) is an approximate method of Bayesian inference originally developed in machine learning (e.g. Jordan et al., 1999; Attias, 2000), finding its way into statistics (e.g. Ormerod and Wand, 2010; Blei et al., 2017) and econometrics (e.g. Koop and Korobilis, 2018; Gunawan et al., 2017). The merit of VB is that it is fast and scalable and therefore useful in big data settings where MCMC might be difficult or infeasible (e.g. forecasting with large vector autoregressions). A drawback of the VB is that its approximation properties are still not well-understood.

There are a number of papers on properties of VB including but not limited to Wang and Titterington (2005, 2006), Hall et al. (2011), Bickel et al. (2013), You et al. (2014), Pati et al. (2017) and Wang and Blei (2018). These papers focus mainly on asymptotic properties of VB. Few analytical results, if any, are available on VB’s finite sample approximation error. Here we study the approximation error of VB for a linear Gaussian multiple equation regression. We focus on vector autoregressions (VAR) since large VARs are of enormous interest in macro-econometric forecasting (e.g. Koop, 2013; Giannone et al., 2015) and VB is likely to be very useful in this context (e.g. Koop and Korobilis, 2018) although our results are valid for any linear Gaussian multi-equation regression.

We derive the divergence between the mean and the mode of VB posterior and those of exact posterior, the magnitude of underestimation in posterior variance and the magnitude of the KL divergence. We show that the approximation error decreases with an increase in sample size and strength of a prior, but increases with the size of the model. We also derive the divergence in the predictive densities and demonstrate that VB estimates predictive densities well. This is important given the enormous interest in forecasting with large VARs for which MCMC is difficult or time consuming.

The paper is organized as follows: Section 2 provides an overview of VB while Section 3 derives the mean-field VB posterior for both conjugate and independent prior VARs. In Section 4, we provide the main results on approximation errors and Section 5 concludes the paper.

2 An Overview of MFVB

Let $\mathbf{y}$ be a vector of observations, $p(\mathbf{y}|\mathbf{\theta})$ the likelihood for a model and $p(\mathbf{\theta})$ a prior where the objective is to learn about the parameter vector $\mathbf{\theta}$ given the observations and the prior. Bayes theorem can be used to derive the posterior kernel easily but learning about $\mathbf{\theta}$ (its posterior density and moments) requires integration of the posterior kernel. For complex models, integration is often done through MCMC but even MCMC can sometimes become intractable (e.g. in big data settings). In VB the idea is to approximate the posterior $p(\mathbf{\theta}|\mathbf{y})$ with a density $q(\mathbf{\theta})$ that is more tractable. The optimal $q(\mathbf{\theta})$ is obtained by minimizing the KL divergence 111For a more detailed review of VB see Ormerod and Wand (2010) or Blei et al. (2017).

[TABLE]

over all $q(\mathbf{\theta})$s that belong to a family of densities $\Lambda.$ A commonly used family includes all $q(\mathbf{\theta})$s that can be written as $q(\mathbf{\theta})=\prod\limits_{k=1}^{M}{{{q}_{k}}(}{{\mathbf{\theta}}_{k}})$ where $\{{{\mathbf{\theta}}_{1}},....,{{\mathbf{\theta}}_{M}}\}$ is some partition of $\mathbf{\theta}$ and the ${{q}_{k}}$s are probability density functions. This factorized form corresponds to an approximation framework known as the mean field; thus, this version of VB is often referred to as mean-field variational Bayes (MFVB).

Using calculus of variations, it has been shown that the optimal ${{q}_{k}}$s are obtainable through the following iterative procedure

[TABLE]

where $p(\mathbf{y},\mathbf{\theta})=p(\mathbf{y}|\mathbf{\theta})p(\mathbf{\theta})$, ${{E}_{{{\theta}_{-k}}}}=\int{\log p(\mathbf{y},\mathbf{\theta}){{\prod\nolimits_{j\neq k}{{{q}_{j}}({{\mathbf{\theta}}_{j}})}}^{old}}}$ and ${{q}_{1}}{{({{\mathbf{\theta}}_{1}})}^{old}},....,{{q}_{M}}{{({{\mathbf{\theta}}_{M}})}^{old}}$ are current estimates for $q$. The iteration is stopped when the increase in $\ln\underline{ML}$ (defined below) becomes negligible. In practice the partition of $\mathbf{\theta}$ is often chosen in a way such that $q_{j}({{\mathbf{\theta}}_{j}})$s are known density functions that depend on the moments of the remaining parameters. As a result, it is often sufficient to iterate over the appropriate moments extracted from the distributions in (2.1).

It can be shown that

[TABLE]

where $ML$ denotes the marginal likelihood. If we define

[TABLE]

then given that $KL(q||p)$ is positive, $\ln\underline{ML}$ provides a lower bound to the log-marginal-likelihood. Note that minimization of $KL(q||p)$ is equivalent to maximization of $\ln\underline{ML}$ therefore $\ln\underline{ML}$ (which can be computed using VB) is often used to monitor the convergence of (2.1).

3 MFVB for Vector Autoregressions

Since the seminal paper of Sims (1980), vector autoregressions have become an essential tool in macroeconomic analysis and forecasting. They are extensions of univariate autoregressive models to a vector of variables 222For a detailed review of Bayesian VARs see e.g. Karlsson (2013) or Koop and Korobilis (2010).. A VAR(d) can be written as

[TABLE]

where $t=1,...,T$ indexes time, $d$ is the lag order, ${{\mathbf{y}}_{t}}$ is an $M\times 1$ vector of observations at time $t$, ${\mathbf{\gamma}}_{0}$ is an $M\times 1$ vector of constants, the ${{\mathbf{\Gamma}}_{i}}$s are $M\times M$ matrices of autoregressive coefficients and ${\mathbf{\varepsilon}}_{t}$ is an $M\times 1$ error vector with mean $\mathbf{0}$ and covariance matrix $\mathbf{\Sigma}$. Define $p=Md+1$ (the number of parameters in each equation), a matrix ${{\mathbf{Y}}_{T\times M}}=({{\mathbf{y}}_{1}},...,{{\mathbf{y}}_{T}}{)}^{\prime}$, a $1\times p$ vector ${{\mathbf{x}}_{t}}=(1,{{\mathbf{{y}^{\prime}}}_{T-1}},...,{{\mathbf{{y}^{\prime}}}_{T-d}})$, ${{\mathbf{X}}_{T\times p}}=({{\mathbf{{x}^{\prime}}}_{1}},...,{{\mathbf{{x}^{\prime}}}_{T}}{)}^{\prime}$, a matrix of coefficients ${{\mathbf{\Gamma}}_{p\,\,\times M}}=({{\mathbf{\gamma}}_{0}},{{\mathbf{\Gamma}}_{1}},...,{{\mathbf{\Gamma}}_{d}}{)}^{\prime}$ and a matrix ${{\mathbf{E}}_{T\times M}}=({{\mathbf{\varepsilon}}_{1}},...,{{\mathbf{\varepsilon}}_{T}}{)}^{\prime}$, then equation (3.1) can be written as:

[TABLE]

where $\mathcal{MN}$ denotes a matricvariate normal distribution (see Appendix-D for a definition). We first consider a natural-conjugate prior where the exact and the VB approximate posterior and the error in the MFVB approximation can be derived analytically.

Normal-Wishart Conjugate Prior: For a VAR model, the natural-conjugate prior is given in the following normal-Wishart form:

[TABLE]

where $\mathbf{\underline{\Gamma}}$ is a $p\times M$ matrix of prior means, $\mathbf{\underline{V}}$ and $\mathbf{\underline{S}}$ are $p\times p$ and $M\times M$ positive-definite matrices, $\mathcal{W}$ and $\underline{\upsilon}$ denote the Wishart distribution and its degrees of freedom. In this case, it is well-known that the exact posterior is a matricvariate normal-Wishart distribution (e.g. Karlsson, 2013):

[TABLE]

Other quantities of interest are marginal posteriors, posterior moments, marginal likelihoods and predictive densities that are available in analytical forms and are given in Table-1.

Now let’s consider MFVB inference where the VB approximate posterior density is factorized into $q(\mathbf{\Gamma},{{\mathbf{\Sigma}}^{-1}})=q(\mathbf{\Gamma})q({{\mathbf{\Sigma}}^{-1}})$. Using (2.1) it can be shown (see the Appendix-A) that the VB posterior is

[TABLE]

without a need for an iterative process !. Related quantities of interest such as marginal posteriors, posterior moments, marginal likelihoods and predictive densities are given in Table-1.

Normal-Wishart Independent Prior: A conjugate prior has the advantage that one can derive the approximation errors analytically. The potential drawback is that the results might be crucially dependent on the conjugacy assumption. Therefore, we also consider a VAR with an independent prior. In this case we cannot derive the approximation error analytically, but we use a real data example to see the extent to which the results differ from the conjugate VAR’s results.

To discuss VAR with an independent prior we appeal to another way of writing a VAR model (see e.g. Koop and Korobilis, 2010):

[TABLE]

$\And\ {{\mathbf{Z}}_{t}}=\left(\begin{matrix}{{{\mathbf{{z}^{\prime}}}}_{1t}}&\mathbf{0}&\cdots&\mathbf{0}\\ \mathbf{0}&{{{\mathbf{{z}^{\prime}}}}_{2t}}&\ddots&\vdots\\ \vdots&\ddots&\ddots&\mathbf{0}\\ \mathbf{0}&\cdots&\mathbf{0}&{{{\mathbf{{z}^{\prime}}}}_{M\,t}}\\ \end{matrix}\right)$ where if $\mathbf{z^{\prime}}_{mt}=\mathbf{x}_{t}$ for $m=1,...,M$, we obtain the same equation as (3.1). In a more compact form:

[TABLE]

where $\mathcal{N}$ denotes a multivariate normal distribution. For the independent case, we assume that the parameter vector $\mathbf{\beta}$ and the precision matrix $\mathbf{\Sigma}^{-1}$ are a priori independent:

[TABLE]

It is well-known that under this prior, the posterior can be obtained through a Gibbs sampler (e.g. Karlsson, 2013) defined by

[TABLE]

Marginal likelihoods and predictive densities are not available in analytical form in this case, but can be obtained through simulation (e.g. Karlsson, 2013).

Now let’s assume $q(\mathbf{\beta},{{\mathbf{\Sigma}}^{-1}})=q(\mathbf{\beta})q({{\mathbf{\Sigma}}^{-1}})$, then MFVB posterior can be obtained using (2.1) as

[TABLE]

where $\{{{\mathbf{\bar{\beta}}}_{q}},{{\mathbf{\bar{V}}}_{q}},\mathbf{\bar{S}}_{q}^{-1}\}$ are obtained by using a starting value for $\overline{\mathbf{\Sigma}_{q}^{-1}}$ and iterating the following until convergence

[TABLE]

$\ln\underline{ML}={{E}_{q}}\ln p(\mathbf{y,\theta})-{{E}_{q}}\ln q(\mathbf{\theta})$, which is used for monitoring of the convergence, can be obtained using (2.3), (A.1), and the entropy of normal and Wishart distributions, as

[TABLE]

where ${\Gamma}_{M}$ is the multivariate gamma function, $\left|\ .\ \right|$ is the determinant and $tr\{.\}$ denotes the trace of a matrix.

While calculating the exact predictive density requires simulation, using arguments similar to those used in Appendix-B, the MFVB predictive density can be obtained as sum of a normal and a t-distribution with the following mean and variance

[TABLE]

4 MFVB Approximation Error

Table-1 compares various features of MFVB and of the exact posterior for the conjugate prior case. As this table shows, the posterior means for both $\mathbf{\Gamma}$ and ${{\mathbf{\Sigma}}^{-1}}$ are equal but MFVB underestimates variances of the parameters, confirming the folklore. The variance of ${{\mathbf{\Gamma}}_{q}}$ is smaller than the variance of $\mathbf{\Gamma}$ by a factor of $1-\frac{M+1}{T+\underline{\upsilon}}$ which is close to one unless the number of equations is large (see Table-2). Variances of elements of $\mathbf{\Sigma}_{q}^{-1}$ are smaller by a factor of $1-({p+1)}/{(T+\underline{\upsilon})}\;$ indicating small underestimation when $p$ (the number of parameters in each equation) is small compared to $T+\underline{\upsilon}$. However, it could be substantial for large VARs (see Table-2). The mode of the approximate posterior is equal to the mode of the exact posterior for $\mathbf{\Gamma}$ but for $\mathbf{\Sigma}_{q}^{-1}$ it is smaller by a factor of $1-{p}/{(T+\underline{\upsilon}+p)}\;$. The KL divergence which is a measure of closeness of the approximate posterior to the exact posterior can be derived as

[TABLE]

and using Stirling’s approximation to gamma functions as

[TABLE]

Note that KL increases as the size of the model (i.e. the numbers of parameters in each equation) increases and decreases with an increase in the number of observations $T$, and with the strength of the prior for ${{\mathbf{\Sigma}}^{-1}}$ (i.e. $\underline{\upsilon}$). The KL does not depend on the data and the other hyper-parameters !

Note that

[TABLE]

where $i,j=1,..,M$ and $h=1,...,Mp$. The first line shows that with an increase in the number of observations, variances of parameters based on MFVB approaches variances of parameters from the exact Bayes. The second line shows the same results with an increase in strength of the prior for $\Sigma^{-1}$.

Also note that the predictive densities based on the exact and MFVB posteriors share the same mean and an almost equal variance. The size of the model does not seem to affect predictive densities (see Tables 1 and 2).

To study the approximation error for the VAR with an independent prior, we estimate a small (M=3) and a medium VAR (with M=7) using the same data set as in Giannone et al. (2015) assuming four lags (d = 4). The data set is based on the benchmark US data set of Stock and Watson (2008) and consists of 200 observations on US macro quarterly variables from the first quarter of 1959 to the last quarter of 2008. The seven variables are real GDP, federal funds rate, GDP deflator, real investment, real consumption, hours worked and real compensation per hour. Hyper-parameters (i.e. $\mathbf{\underline{\Gamma}},\mathbf{\underline{S}},\mathbf{\underline{V}},\underline{\upsilon},\mathbf{\underline{\beta}}$) have been chosen based on the so called Minnesota prior as described in e.g. Karlsson (2013).

Table-3 reports $\ln ML$, $\ln\underline{ML}$ and KL divergence for both the conjugate and independent prior models. As it can be seen, the KL divergences for conjugate and independent VARs, although not equal, are of the same order of magnitude and they both increase with the size of the model. Our other results (not reported here) show that KL divergence decreases with increasing $\underline{\upsilon}$ and $T$.

Since the number of variables in VAR models is large, we cannot present comparisons for each of the parameters here, but Table-4 gives means and variances of the diagonal elements of the precision matrix for the small VAR. The results suggest that MFVB’s mean of ${{\mathbf{\Sigma}}^{-1}}$ is very well estimated but its variance is underestimated and the amount of underestimation is of a similar order to that from the the conjugate case (see Table-6 for the medium VAR’s estimation results).

Table-5 gives the ratios of means and variances from VB to those from the exact (or MCMC) predictive densities for the small VAR [see Table-6 for medium VAR]. For the conjugate case, the means are equal but the variances are smaller for VB, although the difference is negligible. A similar statement can be made for the independent prior case although underestimation seems slightly higher (perhaps more a result of simulation errors).

Tables 3 to 6 suggest VB’s approximation errors derived for the conjugate prior case also holds for the independent prior case although not in an exact manner.

5 Conclusion

We derive analytical expressions for the MFVB’s approximation error for a VAR with a conjugate prior. We also show that errors of a similar order hold for a VAR with independent priors based on a real data example. Our results confirm the claim that MFVB underestimates variances of the parameters and shows that the magnitude of underestimation (and also the KL divergence) change directly with the size of the model but inversely with the number of observations and the strength of the prior for variance. MFVB, at least for standard VARs, is as good as MCMC for estimating posterior means, and for forecasting (our results and Koop and Korobilis, 2018). MFVB also provides an ideal candidate for reciprocal importance sampling (see Hajargasht and Woźniak, 2018). Estimation of variances can also be improved (e.g. Giordano et al., 2017).

Appendix A Derivation of (3.5)

We start with the posterior kernel

[TABLE]

where C is a constant term. Taking expectation with respect to ${{\mathbf{\Sigma}}^{-1}}$ and using the completing of squares trick (see e.g. Karlsson, 2013) we can show:

[TABLE]

Similarly, after taking expectations with respect to $\mathbf{\Gamma}$ we have

[TABLE]

[TABLE]

where ${{\mathbf{\Omega}}_{ij}}=tr\left[({{{\mathbf{\underline{V}}}}^{-1}}\mathbf{+{X}^{\prime}X})\operatorname{cov}({{\mathbf{\Gamma}}_{i}},{{\mathbf{\Gamma}}_{j}})\right]$ and ${\mathbf{\Gamma}}_{i}$ and ${\mathbf{\Gamma}}_{j}$ are $i\text{-}th$ and $j\text{-}th$ columns of $\mathbf{\Gamma}$. To derive this we have used some results on expectations from Appendix-D.

Normally, appropriate moments from (A.2) and (A.3) are iterated until convergence and then $q(\mathbf{\Gamma})q({{\mathbf{\Sigma}}^{-1}})$ is treated as the posterior but note from (A.2) that we can write

[TABLE]

where $\bar{\sigma}_{ij}^{+}$s are elements of matrix ${{\overline{{{\mathbf{\Sigma}}^{-1}}}}^{\,-1}}$.  Therefore

[TABLE]

where the last equality comes from ${{\mathbf{\Sigma}}^{-1}}\sim{\ }\mathcal{W}({{\mathbf{\bar{S}}}_{q}}^{-1},{{\bar{\upsilon}}_{q}}).$ Substituting $\mathbf{\Omega}$ from (A.4) in (A.3) we obtain:

[TABLE]

Note that starting with (A.2) then (A.6) and (A4), $q(\mathbf{\Gamma})$ and $q({{\mathbf{\Sigma}}^{-1}})$ can be computed separately without a need for an iterative scheme.

Appendix B Derivation of $q(\mathbf{y}_{T+1}|\mathbf{Y})$

Start with ${{\mathbf{y}}_{T+1}}={{\left(\mathbf{x}_{T+1}\mathbf{\Gamma}\right)}^{\prime}}+{{\mathbf{\varepsilon}}_{T+1}}$ then using (3.5) and (D.1) we have

[TABLE]

Using (3.5) and (D.4)

[TABLE]

using (D.4) again gives

[TABLE]

Using (D.1) and (D.4) one more time:

[TABLE]

[TABLE]

Appendix C Derivation of Posterior Modes

(i) Conjugate VAR: Note that log-posterior can be written as

[TABLE]

Maximize this with respect to $(\mathbf{\Gamma},{{\mathbf{\Sigma}}^{-1}})$

[TABLE]

[TABLE]

where ($\mathbf{\hat{\Gamma}}$, ${{\mathbf{\hat{\Sigma}}}^{-1}}$) denotes the mode of posterior. To derive these results we have used some matrix derivative formulas from the Appendix-D.

(ii) Independent Prior VAR

Log-posterior:

[TABLE]

Maximize this with respect to $(\mathbf{\Gamma},{{\mathbf{\Sigma}}^{-1}})$

[TABLE]

[TABLE]

To derive these results, we have used some matrix derivative formulas from Appendix-D. The mode of the exact posterior i.e. [$\mathbf{\hat{\beta}}$,${{\mathbf{\hat{\Sigma}}}^{-1}}$] can be obtained by iterating over equations (A.7) and (A.8). In contrast, the mode of the MFVB posterior can be obtained (using 3.9 and known results on modes of normal and Wishart distributions) by iterating over the following equations

[TABLE]

where $\lambda=1+\dfrac{M+1}{T+\underline{\upsilon}-M-1}$ .

Appendix D Distributions, Matrix Derivatives and Expectations

**Distributions:

**Matricvariate Normal Distribution: if ${{\mathbf{X}}_{pm}}\sim{\ }\mathcal{MN}\{\mathbf{\bar{X}},\mathbf{\Sigma},\mathbf{V}\}$ then

[TABLE]

Wishart Distribution: if $\mathbf{\Sigma}\sim{\ }\mathcal{W}\{\mathbf{S},\upsilon\}$ then

[TABLE]

Matricvariate t-distribution: if ${{\mathbf{X}}_{pq}}\sim{\ }\mathcal{MT}\{\mathbf{\bar{X}},{{\mathbf{S}}_{q}},{{\mathbf{V}}_{p}},\upsilon\}$ then

[TABLE]

Multivariate t-distribution: if ${{\mathbf{x}}_{q\times 1}}\sim{\ }\mathcal{T}\{\mathbf{x},\mathbf{S},\upsilon\}$ then

[TABLE]

Some Matrix Derivatives:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Some Results on Expectations:

      Let $E(\mathbf{x})=\mathbf{\bar{x}}$ and $Var(\mathbf{x})=\mathbf{\Sigma}$ then

[TABLE]

[TABLE]

[TABLE]

[TABLE]

where ${{\mathbf{\Omega}}_{ij}}=tr\{Cov({{\mathbf{x}}_{i}},{{\mathbf{x}}_{j}})\mathbf{A}\}$, ${{\mathbf{x}}_{i}}$ and ${{\mathbf{x}}_{j}}$ are i-th and j-th columns of $\mathbf{X}$ for $i,j=1,...,q$.