# Mapping multi-modal dynamic network activity during naturalistic music listening

## TL;DR

This paper introduces a new method to study brain activity during naturalistic music listening by combining brain, behavior, and stimulus data.

## Contribution

The novel workflow integrates multi-modal dynamic data using HMM and PLS for interpreting brain-behavior-stimulus interactions.

## Key findings

- HMM and PLS can jointly model EEG, behavioral, and stimulus data during music listening.
- The workflow reveals dynamic relationships between brain states and external stimuli.
- Current tools have limitations in modeling complex, real-world multi-modal interactions.

## Abstract

The human brain is a complex, adaptive system capable of parsing complex stimuli and generating complex behaviour. Understanding how to model and interpret the dynamic relationship between brain, behaviour, and the environment will provide vital information on how the brain responds to real-world stimuli, develops and ages, and adapts to pathology. Modelling together numerous streams of dynamic data, however, presents sizable methodological challenges. In this paper, we present a novel workflow and sample interpretation of a data set incorporating brain, behavioural, and stimulus data from a music listening study. We use hidden Markov modelling (HMM) to extract state time series from continuous high-dimensional EEG and stimulus data, estimate time series variables consistent with HMM from continuous low-dimensional behavioural data, and model the multi-modal data together using partial least squares (PLS). We offer a sample interpretation of the results, including a discussion on the limitations of the currently available tools, and discuss future directions for dynamic multi-modal analysis focusing on naturalistic behaviours.