# Maximizing the accuracy of genetic variance estimation and using a novel generalized effective sample size to improve simulations

## TL;DR

This paper introduces a more accurate method for estimating genetic variance and a new effective sample size to improve genetic simulations for breeding programs.

## Contribution

A novel PEV-based variance estimation and generalized effective sample size for more accurate and flexible genetic simulations.

## Key findings

- PEV-based additive variance estimation significantly reduces root mean square errors compared to traditional methods.
- The generalized effective sample size improves simulation accuracy by accounting for sampling variation.
- The method supports complex interactions like genotype by environment effects in breeding program simulations.

## Abstract

We developed an improved variance estimation that incorporates prediction error variance as a correction factor, alongside a novel generalized effective sample size to enhance simulations. This approach enables precise control of variance components, accommodating for more flexible and accurate simulations.

Phenotypic variation in field trials results from genetic and environmental factors, and understanding this variation is critical for breeding program simulations. Additive genetic variance, a key component, is often estimated using linear mixed models (LMM), but can be biased due to improper scaling of the genomic relationship matrix. Here, we show that this bias can be minimized by incorporating prediction error variance (PEV) as a correction factor. Our results demonstrate that the PEV-based estimation of additive variance significantly improves accuracy, with root mean square errors orders of magnitude lower than traditional methods. This improved accuracy enables more realistic simulations, and we introduce a novel generalized effective sample size (ESS) to further refine simulations by accounting for sampling variation. Our method outperforms standard simulation approaches, allowing flexibility to include complex interactions such as genotype by environment effects. These findings provide a robust framework for variance estimation and simulation in genetic studies, with broad applicability to breeding programs.

The online version contains supplementary material available at 10.1007/s00122-025-04861-8.