Interpretation and approximation tools for big, dense Markov chain transition matrices in ecology and evolution

TL;DR

This paper introduces methods to interpret and approximate large, dense Markov chain transition matrices in ecology and evolution, enabling their use despite computational and interpretative challenges.

Contribution

It presents a novel approach combining network analysis and sparse approximation algorithms to handle big matrices while preserving key properties.

Findings

Transform big matrices into interpretable graphs using network analysis

Reduce memory usage by 90% with minimal information loss

Maintain over 99% of the dominant eigenvector information

Abstract

Markov chains are a common framework for individual-based state and time discrete models in ecology and evolution. Their use, however, is largely limited to systems with a low number of states, since the transition matrices involved pose considerable challenges as their size and their density increase. Big, dense transition matrices may easily defy both the computer's memory and the scientists' ability to interpret them, due to the very high amount of information they contain; yet approximations using other types of models are not always the best solution. We propose a set of methods to overcome the difficulties associated with big, dense Markov chain transition matrices. Using a population genetic model as an example, we demonstrate how big matrices can be transformed into clear and easily interpretable graphs with the help of network analysis. Moreover, we describe an algorithm to save computer memory by substituting the original matrix with a sparse approximate while preserving all its mathematically important properties. In the same model example, we manage to store about 90% less data while keeping more than 99% of the information contained in the matrix and a closely corresponding dominant eigenvector. Our approach is an example how numerical limitations for the number of states in a Markov chain can be overcome. By facilitating the use of state-rich Markov chain models, they may become a valuable supplement to the diversity of models currently employed in biology.