Matrix dissimilarities based on differences in moments and sparsity

TL;DR

This paper introduces a novel dissimilarity measure based on differences in moments and sparsity, providing deeper insights into group differences across various biological and social datasets.

Contribution

The paper presents a new dissimilarity approach that captures key factors like moments and sparsity, enhancing analysis of complex data beyond traditional methods.

Findings

Sparsity dissimilarity is as effective as mean dissimilarity in predicting COVID-19 drug effects.

The method reveals underlying biological factors such as gene regulation and heterogeneity.

Extensive dataset reanalysis demonstrates the approach's broad applicability.

Abstract

Generating a dissimilarity matrix is typically the first step in big data analysis. Although numerous methods exist, such as Euclidean distance, Minkowski distance, Manhattan distance, Bray Curtis dissimilarity, Jaccard similarity and Dice dissimilarity, it remains unclear which factors drive dissimilarity between groups. In this paper, we introduce an approach based on differences in moments and sparsity. We show that this method can delineate the key factors underlying group differences. For example, in biology, mean dissimilarity indicates differences driven by up down regulated gene expressions, standard deviation dissimilarity reflects the heterogeneity of response to treatment, and sparsity dissimilarity corresponds to differences prompted by the activation silence of genes. Through extensive reanalysis of genome, transcriptome, proteome, metabolome, immune profiling, microbiome, and social science datasets, we demonstrate insights not captured in previous studies. For instance, it shows that the sparsity dissimilarity is as effective as the mean dissimilarity in predicting the alleviation effects of a COVID 19 drug, suggesting that sparsity dissimilarity is highly meaningful.