# Bi-cross validation for estimating spectral clustering hyper parameters

## TL;DR

This paper demonstrates that bi-cross validation can be effectively used to estimate the number of clusters and hyperparameters in spectral clustering, especially for large-scale x-ray scattering data, improving anomaly detection.

## Contribution

It introduces a novel application of bi-cross validation to jointly estimate spectral clustering parameters and the number of clusters, enhancing analysis of large-scale scientific data.

## Key findings

- BCV maps to spectral clustering hyperparameter estimation
- Enables identification of dropped shots in LCLS data
- Facilitates detection of rare and anomalous events

## Abstract

One challenge impeding the analysis of terabyte scale x-ray scattering data from the Linac Coherent Light Source LCLS, is determining the number of clusters required for the execution of traditional clustering algorithms. Here we demonstrate that previous work using bi-cross validation (BCV) to determine the number of singular vectors directly maps to the spectral clustering problem of estimating both the number of clusters and hyper parameter values. These results indicate that the process of estimating the number of clusters should not be divorced from the process of estimating other hyper parameters. Applying this method to LCLS x-ray scattering data enables the identification of dropped shots without manually setting boundaries on detector fluence and provides a path towards identifying rare and anomalous events.