A Penalized Multi-trait Mixed Model for Association Mapping in Pedigree-based GWAS

TL;DR

This paper introduces a penalized multivariate linear mixed model for GWAS that effectively captures correlations among multiple traits, improves variable selection, and enhances prediction accuracy in pedigree-based studies.

Contribution

It develops a novel penalized-MTMM that models both within- and between-trait variance components, integrating variable selection with mixed models for multi-trait GWAS.

Findings

The proposed model outperforms uni-trait methods in simulations.

It accurately identifies genetic markers associated with multiple traits.

The approach demonstrates improved prediction performance on real GWAS data.

Abstract

In genome-wide association studies (GWAS), penalization is an important approach for identifying genetic markers associated with trait while mixed model is successful in accounting for a complicated dependence structure among samples. Therefore, penalized linear mixed model is a tool that combines the advantages of penalization approach and linear mixed model. In this study, a GWAS with multiple highly correlated traits is analyzed. For GWAS with multiple quantitative traits that are highly correlated, the analysis using traits marginally inevitably lose some essential information among multiple traits. We propose a penalized-MTMM, a penalized multivariate linear mixed model that allows both the within-trait and between-trait variance components simultaneously for multiple traits. The proposed penalized-MTMM estimates variance components using an AI-REML method and conducts variable selection and point estimation simultaneously using group MCP and sparse group MCP. Best linear unbiased predictor (BLUP) is used to find predictive values and the Pearson's correlations between predictive values and their corresponding observations are used to evaluate prediction performance. Both prediction and selection performance of the proposed approach and its comparison with the uni-trait penalized-LMM are evaluated through simulation studies. We apply the proposed approach to a GWAS data from Genetic Analysis Workshop (GAW) 18.