# Crafted experiments to evaluate feature selection methods for single-cell RNA-seq data

## TL;DR

This paper introduces a new approach for evaluating gene selection methods in single-cell RNA-seq data using crafted experiments and a new method called GOF.

## Contribution

The paper introduces a novel framework for benchmarking gene selection methods using crafted experiments and a new univariate distribution-oriented method called GOF.

## Key findings

- Crafted experiments effectively evaluate feature selection methods by perturbing real datasets.
- GOF performs well on both crafted and real datasets, selecting features that robustly identify biological signals.
- Different GOF methods perform best in specific contexts, as shown through varied crafting approaches.

## Abstract

While numerous methods have been developed for analyzing scRNA-seq data, benchmarking various methods remains challenging. There is a lack of ground truth datasets for evaluating novel gene selection and/or clustering methods. We propose the use of crafted experiments, a new approach based upon perturbing signals in a real dataset for comparing analysis methods. We demonstrate the effectiveness of crafted experiments for evaluating new univariate distribution-oriented suite of feature selection methods, called GOF. We show GOF selects features that robustly identify crafted features and perform well on real non-crafted data sets. Using varying ways of crafting, we also show the context in which each GOF method performs the best. GOF is implemented as an open-source R package and freely available under GPL-2 license at https://github.com/siyao-liu/GOF. Source code, including all functions for constructing crafted experiments and benchmarking feature selection methods, are publicly available at https://github.com/siyao-liu/CraftedExperiment.