Hierarchical Clustering for Finding Symmetries and Other Patterns in Massive, High Dimensional Datasets

TL;DR

This paper explores hierarchical clustering techniques to identify symmetries and patterns in large, high-dimensional datasets, emphasizing their role in uncovering invariants and intrinsic data properties.

Contribution

It reviews the theoretical foundations of hierarchy, including ultrametric topology and algebraic structures, and demonstrates their application in analyzing complex real-world data.

Findings

Hierarchical clustering reveals symmetries in data.

Application in chemistry and finance shows effectiveness.

Hierarchies help identify invariants and intrinsic properties.

Abstract

Data analysis and data mining are concerned with unsupervised pattern finding and structure determination in data sets. "Structure" can be understood as symmetry and a range of symmetries are expressed by hierarchy. Such symmetries directly point to invariants, that pinpoint intrinsic properties of the data and of the background empirical domain of interest. We review many aspects of hierarchy here, including ultrametric topology, generalized ultrametric, linkages with lattices and other discrete algebraic structures and with p-adic number representations. By focusing on symmetries in data we have a powerful means of structuring and analyzing massive, high dimensional data stores. We illustrate the powerfulness of hierarchical clustering in case studies in chemistry and finance, and we provide pointers to other published case studies.