# Comparing human vs. machine-assisted analysis to develop a new approach for Big Qualitative Data Analysis

## TL;DR

This study compares human and machine-assisted analysis of large qualitative healthcare data, finding that AI tools can speed up analysis but require human input for nuanced insights.

## Contribution

The paper introduces a hybrid human-machine approach for Big Qualitative Data Analysis and identifies limitations in current AI tools for inductive qualitative research.

## Key findings

- Machine-assisted analysis reduced analysis time significantly, especially for theme identification.
- Human analysis identified an emergent 'ambiguous' sentiment category not captured by AI tools.
- Hybrid approaches combining AI efficiency with human nuance are effective for Big Qualitative Data.

## Abstract

The exponential growth of Big Qualitative (Big Qual) data in healthcare research presents methodological challenges for traditional analysis approaches. This study evaluates the effectiveness of machine-assisted analysis using artificial intelligence (AI) tools compared to human-only analysis for processing large-scale qualitative datasets, using the Royal College of Anaesthetists’ 7th National Audit Project (NAP7) baseline survey as a test case.

We conducted a comparative methodological study analysing 5,196 free-text responses about peri-operative cardiac arrest experiences. Three researchers established a human-coded reference standard following SRQR guidelines. We then applied machine-assisted analysis using Pulsar for exploratory analysis and Caplena for sentiment and thematic analysis, evaluating performance against the human gold standard using STARD-AI reporting standards. Performance metrics included accuracy, precision, recall, F1-scores, and Cohen’s Kappa, with confidence intervals calculated using bootstrap resampling.

Machine-assisted analysis substantially reduced analysis time, with particularly dramatic improvements in theme identification speed. The machine-assisted approach achieved good thematic and sentiment classification accuracy compared to the human reference standard, though human analysis identified an emergent ‘ambiguous’ sentiment category that current AI tools cannot accommodate, highlighting limitations in commercial platforms’ flexibility for inductive analysis.

Machine-assisted analysis offers substantial efficiency gains with acceptable accuracy trade-offs for large-scale qualitative data analysis. However, human expertise remains essential for capturing nuanced meanings, identifying emergent categories, and providing domain-specific interpretation. This hybrid approach represents a viable methodology for Big Qual research, though current AI tools’ constraints in accommodating emergent classification schemes remain a limitation. Our findings establish benchmarks for future development of more flexible AI systems adapted to qualitative research paradigms.

The use of Artificial intelligence (AI) in health research has grown over recent years. However, analysis of large qualitative datasets known as Big Qualitative Data, in public health using AI, is a relatively new area of research. Here, we use novel techniques of human-machine learning known as ‘machine-assisted’ analysis and natural language processing where computers learn how to handle and interpret human language, to analyse a large national survey. The Royal College of Anaesthetists’ 7th National Audit Project is a large UK-wide initiative examining peri-operative cardiac arrest. We use the free-text data from this survey to test and validate our novel methods and compare analysing the data by hand (human) vs. machine-assisted analysis. Using two AI tools to conduct the analysis we found that the machine-assisted analysis significantly reduced the time to analyse the dataset. Extra human input, however, was required to provide topic expertise and nuance to the analysis. The AI tools reduced the sentiment analysis to positive, negative or neutral, but the human input was able to identify a fourth ‘ambiguous’ category. The insights gained from this approach present ways that AI can help inform targeted interventions and quality improvement initiatives to enhance patient safety, in this case, in peri-operative cardiac arrest management.