Method of fractal diversity in data science problems

TL;DR

This paper introduces a fractal-based method to measure cross-correlation in data, applicable to large, non-Gaussian datasets, with potential uses in identifying biologically active conformers in X-ray spectra.

Contribution

The paper presents a novel fractal diversity method for quantifying data cross-correlation that is invariant under linear transformations and applicable to complex data sets.

Findings

The method effectively distinguishes Gaussian from non-Gaussian data.

It successfully identifies biologically active conformers in X-ray diffraction spectra.

The approach is universal and independent of data distribution or correlation nature.

Abstract

The parameter (SNR) is obtained for distinguishing the Gaussian function, the distribution of random variables in the absence of cross correlation, from other functions, which makes it possible to describe collective states with strong cross-correlation of data. The signal-to-noise ratio (SNR) in one-dimensional space is determined and a calculation algorithm based on the fractal variety of the Cantor dust in a closed loop is given. The algorithm is invariant for linear transformations of the initial data set, has renormalization-group invariance, and determines the intensity of cross-correlation (collective effect) of the data. The description of the collective state is universal and does not depend on the nature of the correlation of data, nor is the universality of the distribution of random variables in the absence of data correlation. The method is applicable for large sets of non-Gaussian or strange data obtained in information technology. In confirming the hypothesis of Koshland, the application of the method to the intensity data of digital X-ray diffraction spectra with the calculation of the collective effect makes it possible to identify a conformer exhibiting biological activity.