Coupled-oscillator-humanizer revealed possible ensemble players’ ability to discriminate cross-correlation structures in auditory sequences of paired drum tapping

TL;DR

The study shows that people with ensemble music experience can detect subtle timing patterns in drum tapping that reflect coordination between players.

Contribution

The research introduces a novel method to assess listeners' ability to perceive cross-correlation structures in ensemble-like timing sequences.

Findings

Participants with ensemble experience could distinguish cooperative timing patterns from random ones.

Musical sophistication index did not correlate with the discrimination ability.

The results suggest that ensemble experience enhances perception of coordination in timing sequences.

Abstract

Natural fluctuations in the timing sequence are essential for a dynamic and expressive rendition of music. Various studies have demonstrated the statistical structure of such timing fluctuations in solo music performances and listeners’ ability to perceive them. However, the listening ability of timing structures in ensemble performances involving multiple players remains unclear despite its importance in understanding actual music skills. Here, we assessed listeners’ ability to distinguish the statistics of timing variations that resembled mutual adaptations between a pair of tapping players as a simplified example of ensemble performance. We created sound stimuli in which the drum-tapping timing sequences fluctuated either cooperatively among the pair or randomly using a generative model for synchronized pair tapping. Listing tests to discriminate between these stimuli, and a questionnaire on music experience and sophistication were conducted as a web-based experiment. Consequently, a trend was observed where participants with ensemble experience could discriminate between stimuli, whereas participants without ensemble experience could not. This discrimination ability did not correlate with the musical sophistication index. These results suggest that listeners with a certain music experience, particularly those involved in ensemble performances, can perceive the individuality of each player and the coordination between them based on the timing sequence structure.

Figures

Overview of the methods and processes for generating experimental stimuli.(A) HUM stimuli were generated from the model with a consistent cross-correlation structure, and based on these HUM stimuli, RAN stimuli with a random cross-correlation structure were produced. These stimuli were sequentially presented to participants, who were asked to discriminate between them. (B) Correlation structures of the stimuli.

Comparison of the percentage of “human-like” responses by group and stimulus.chance levels Panels A and B display the data for participants with ensemble experience, whereas panels C and D show the data for participants without ensemble experience. (A) and (C): box charts illustrating the percentages of participants’ responses as “human-like” to the RAN (blue) and HUM (red) stimuli, with adjacent dots representing individual participant data. (B) and (D): box chart for the difference between the percentage of “human-like” responses to RAN and HUM, with neighboring dots indicating each participant’s data. The dashed lines indicate chance levels (50% for panels A and B, and 0 for C and D).

Comparison of sensitivity and response bias.Panels A and B present the data for participants with ensemble experience, whereas panels C and D present the data for those without ensemble experience. (A) and (C): box charts for sensitivity d’, with adjacent dots representing individual participant data. (B) and (D): box charts for bias C, with neighboring dots indicating each participant’s data.

Comparison of accurate response rate in practice trials.Panels A and B present data for participants with and without ensemble experience, respectively, with adjacent dots representing individual participant data.

Equations

Full text

Introduction

In musical performances, mechanical precision similar to that of a metronome is not always desirable. This tendency is evident in Western classical music performances, particularly within the Romantic repertoire. The intentional manipulation of timing by a performer for expressive purposes is termed expressive timing, with prior research identifying characteristic strategies such as phrase-final lengthening [1–3] and phrase-arching [4,5]. Several studies have examined expressive timing in ensemble performances and revealed techniques such as primary voice leading and melody lead, as well as their effects on auditory impressions [1,6,7]. In contrast to Western classical music, the rock and pop genres are typically performed isochronously. Even in such genres, expressive or groovy performance requires micro-fluctuations in timing (micro-timing), i.e., slight deviations from the exact notated (or quantized) timing [8–11]. This timing fluctuation has attracted attention in performance research on various music genres [12–17]. Even in computer-based plays or drum machines, a certain amount of fluctuation is often added to the note timing to mimic human performance, and such programs are called “humanizers” [18]. The present study aims to extend the theoretical framework of the dynamical properties (time evolution structures of a system that cannot be captured simply by using the mean or variance of the fluctuation) of micro-timing in isochronous genres to ensemble situations by integrating it with the theory of coordination dynamics between partners, previously explored in paired tapping tasks. Furthermore, we investigated whether humans possess the perceptual ability to discriminate between coordination dynamics.

Intrinsic and natural fluctuations in musical performance have been widely observed in human cognition and behavior, and have been associated with the ‘ $[eqn]$ ’ statistical structure [19–26]. The $[eqn]$ is a characteristic of the spectral pattern, which reflects the randomness and unpredictability of the fluctuations [24]. The β represents a linear slope of the log-log power spectrum, where β = 0 corresponds to white noise and β = 1 to pink noise [22,27]. Interestingly, listeners prefer music renditions with fluctuating structures closer to pink noise than to white noise [18,28]. This suggests that we can discriminate, to some extent, the $[eqn]$ structure that the auditory series contains.

Because the $[eqn]$ structure is an indicator that characterizes a single time series, it was applied to model the fluctuations in the solo performance. However, for the ensemble performance, the characteristics of the fluctuation structures remain unknown despite several related studies. For example, Hennig (2014) modeled ensemble performance as a mutually interacting complex system (MICS) and demonstrated that this model generated a naturally sounding “Billie Jean” performance [29]. This was achieved by simulating timing fluctuations through a synchronized tapping task in pairs, akin to a coupled oscillator with $[eqn]$ noise. Konvalinka et al. (2010) investigated rhythmic coordination in paired synchronized tapping tasks, finding that inter-tap interval (ITI) time series displayed a characteristic coordination dynamics in which players alternated between long and short intervals by referencing each other’s previous taps, resulting in positive lag ± 1 and negative lag 0 cross-correlation coefficients [30]. Okano et al. (2017) and Okano et al. (2019) replicated these previous findings across various tempos and observed a consistent pattern in ensemble coordination, even when tapping periods were extended [31,32]. These studies provide insights into the complex interplay between timing and coordination in ensemble performances and contribute to our understanding of how these dynamics are perceived and maintained. Okano et al. (2019) demonstrated that simulations of their model reproduced coordination dynamics [31]. Their model was formulated as a coupled oscillator model with a hybrid continuous-time/discrete-reset structure and had four parameters for each partner: phase and period correction parameters a and b, respectively; rebound strength k to an initial tempo; and timing noise σ (see Methods). They also demonstrated that their model replicates the multi-scale dynamics of the fluctuation and coordination of the partners’ ITIs and that these dynamics are modulated by a period-correction parameter (b). This appears to reflect the individuality of coordination among partners [31].

If the timing fluctuations and coordination of partners in an ensemble exhibit individuality and preference, as in the case of solos, humans should be able to discriminate between them. The perception of statistical structures in timing sequences may be affected by the ability to perceive rhythmic sounds and/or prior experience with such sound patterns. Recent studies on human rhythmic coordination suggest that humans can not only perceive and realize local beat synchronization but also adjust the global (dynamical) properties of variations, such as the $[eqn]$ . of the $[eqn]$ structure, in behavioral rhythms to match external rhythms [33–41]. If, as discussed above, the dynamical properties of fluctuations are linked to the perception of performance, the ability to discern how these fluctuations are coordinated among partners would be crucial for an ensemble’s success.

What mechanisms are responsible for these perceptual abilities? One potential explanation for this is the accurate detection of timing fluctuations and asynchrony. However, previous research suggests that listeners do not perceive fluctuations or asynchrony with sufficient accuracy. The just noticeable difference for sound onset interval variations in isochronous sequences is reported to be approximately 6–10 ms for sequences with 240 ms intervals, and approximately 2.5 to 5% for sequences with lger intervals [42,43]. In addition, in the perception of asynchrony, the temporal order becomes imperceptible for time differences smaller than 15 to 35 ms [44,45]. These findings highlight the need to consider alternative mechanisms without presuming the precise detection of local fluctuations and asynchronies.

In this regard, the dynamical systems approach offers a compelling framework by treating an ensemble as a collective unit rather than as a sum of individuals. This approach suggests that ensemble coordination arises from anticipation and adaptation, grounded in self-other integration, and the collective behavior of the entire group, rather than relying on pairwise perceptual information processing [46–48]. Research adopting this approach has identified multiple factors influencing the coordination of performance, including the performer’s intrinsic rhythms [49], visual cues from bodily movements [50–52], cognitive chunking [53] and social relationships [54]. In other words, the dynamical characteristics of an ensemble are perceived through its overall impression, reflecting the collective behavior of members, rather than through a detailed analysis of timing from moment to moment.

This study performed an online listening experiment to examine whether listeners could distinguish between paired ITI series with varying parameter settings as generated by Okano et al. ’s (2019) coupled oscillator model. Our purpose was not to assess whether the generated stimuli sounded “human-like,” but rather to determine whether humans could differentiate coordination structures. Additionally, we explored the relationships among discrimination ability, musical sophistication, and experience in ensemble participation. Although the auditory stimuli presented in the experimental task–paired drum tapping at almost equal intervals–lack variations in pitch, timbre, or amplitude, they capture the core rhythmic element of timing differences and adjustments between performers that arise in a duet, the minimum unit of an ensemble. Timing is an extremely important element in acoustic communication, including music [42]; even a difference of a few tens of milliseconds can affect perceptual quality [13,16]. Thus, investigating the relationship between the auditory discriminability of coordination dynamics and listening impressions using this model serves as an essential early step towards a scientific understanding of the micro-timing of ensembles in the isochronous genre of music.

Methods

Overview

We conducted web-based listening experiments to assess the discrimination of the timing sequence stimuli of paired ITIs generated by the numerical simulation of synchronized tapping (Fig 1) using our previously reported model [31,55]. Three stimulus sequences were generated: humanized (HUM), randomized (RAN), and isochronous (ISO). Before starting the listening test, we presented HUM stimuli as examples of sounds containing “human-like” fluctuations, but RAN and ISO stimuli for showing “not human-like” fluctuation examples. After a brief practice session, the participants joined the test session and judged whether the presented stimulus was “human-like” or “not human-like.” Statistical analyses centered on the percentage of the responses that participants judged HUM and RAN stimuli to be “human-like” and musical experience.

Overview of the methods and processes for generating experimental stimuli.(A) HUM stimuli were generated from the model with a consistent cross-correlation structure, and based on these HUM stimuli, RAN stimuli with a random cross-correlation structure were produced. These stimuli were sequentially presented to participants, who were asked to discriminate between them. (B) Correlation structures of the stimuli.

The cross-correlation structure of the stimuli was examined using windowed detrended cross-correlation (WDCC) analysis [56]. While previous studies on solo performance and Hennig (2014) emphasized on the $[eqn]$ structure, the present study prioritized the cross-correlation structure because controlling stimulus properties by focusing on the $[eqn]$ structure should extend stimuli excessively. It is known that, in estimating $[eqn]$ structures, significant estimation errors arise unless there are at least 256 taps (data points) [27]. Several studies have suggested that WDCC is advantageous for investigating synchronization processes [31,34,56].

Stimuli

Participants listened to the auditory stimuli using a web browser. The stimuli comprised mp3 audio files (S3 File), each containing 36 synchronized drum-tapping sounds produced by a pair of virtual players. In one experiment, participants were exposed to 20 HUM, 20 RAN, and 10 ISO stimuli throughout the example, practice, and test phases. All the stimuli were generated using MATLAB R2023a (MathWorks, USA). The stimuli were prepared according to the following procedure.

Humanized sequence (HUM).

Tap timing series were generated using the numerical simulation of the coupled oscillator model of Okano et al. (2019). In the model, the timing adjustment process in the paired synchronous tapping task was formulated as follows [31,57,55]:

$$[eqn]$$ $$[eqn]$$

for $[eqn]$ , where $[eqn]$ is the time, $[eqn]$ and $[eqn]$ are the phase and angular velocity of the $[eqn]$ -th participant’s tapping, and $[eqn]$ is the zero-mean Gaussian white noise of unit intensity, and $[eqn]$ is the strength of the noise. When the $[eqn]$ -th participant taps (i.e., $[eqn]$ ), the $[eqn]$ -th participant’s phase $[eqn]$ is reset to 0, and the other ( $[eqn]$ -th) participant’s phase $[eqn]$ and angular velocity $[eqn]$ are reset as follows:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ and $[eqn]$ are response functions to modulate the phase and the angular velocity, respectively, and $[eqn]$ and $[eqn]$ are gains for each function. The third term of eq (4) represents the intention to maintain the initial tempo, where $[eqn]$ is its gain.

We used the model parameters of a = 0.3, k = 0.3b, and σ = 0.3 for both players 1 and 2, with variations in b set at 1.5, 1.0, and 0.5 (these variations corresponded to the three participant recruitment announcements). These choices were made because the WDCC structure empirically depends on b: concretely, increasing b deepens the valley of lag-0 WDCC and raises the peak of lag ± 1 WDCC [31]. Based on this property, b = 1.5 was set as a relatively strong level of period correction, which generally reproduces the most prominent WDCC structures, as observed by Okano et al. (2019). The other two levels (b = 1.0 and 0.5) were set to represent moderate and weak periodic corrections, respectively (see Supplementary Table S1 in S1 Appendix A). σ and k modulated the variability of the ITI and asynchronies: these settings were set to produce a level of variability that would not allow a clear perception of the order of the two tap timings or the lengthening or shortening of the ITI (see Table 1). The resulting ITI series pairs were subjected to WDCC analysis to obtain the cross-correlation structure from lag −10 to +10. The generation of ITI series pairs was repeated 1000 times, and the 20 pairs with WDCC structures closest to the mean of the 1000 repetitions were selected as HUM stimuli and as the basis for RAN stimuli (Fig 1A).

Table 1: ITI and asynchrony for each stimulus group.

Randomized sequence (RAN).

The RAN stimuli were generated from the HUM stimuli by shuffling the mean and standard deviation of the asynchronies between players in the HUM and RAN stimuli as follows:

(1) The nth asynchrony in the asynchrony series a = (a1, a2, …) was calculated using the following equation (hereafter, the bold type denotes a vector):

$$[eqn]$$

where $[eqn]$ and $[eqn]$ represent the nth tap timing series for Players 1 and 2, respectively.

(2) The average tap timing series m = (m1, m2, …) between the players was computed as follows:

$$[eqn]$$

where mn represents the nth average tap timing between players.

(3) The randomized tap timing series $[eqn]$ and $[eqn]$ for Players 1 and 2, respectively, were defined as follows:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ represents a randomly sorted halved asynchrony series: for example, $[eqn]$ .

(4) The first-order differences of $[eqn]$ were calculated to obtain a randomized ITI series pair, which was input into the WDCC to derive the cross-correlation structure from lag –10 to lag + 10.(5) Steps (3) and (4) were repeated 1000 times, and the 20 pairs of time-series $[eqn]$ and $[eqn]$ whose cross-correlation structure was closest to the average of the 1000 iterations were selected as the RAN stimuli. The cross-correlation structures of HUM and RAN used in the experiments are illustrated in Fig 1B. The WDCC coefficients for lag –1 to +1 in the resulting RAN stimuli are presented in Table S2 in S1 Appendix A.

Isochronous sequence (ISO).

The ISO stimuli were designed to be easily distinguishable from HUM and RAN, with their ITI fluctuations maintained below the perceptual threshold, and the partner’s taps were perfectly synchronized. This stimulus was used to identify participants who were not paying attention during the listening test. The generation method was similar to that of the RAN, except that the model parameter σ was set to σ = 0.01. The descriptive statistics for the mean and standard deviation of the ITI and asynchrony for HUM, RAN, and ISO are presented in Table 1. Note that the mean ITI of ISO was slightly longer than those of HUM and RAN because the degree of joint rushing decreased when the other parameters were fixed and sigma was reduced. This discrepancy between the ISO and the other stimuli was not problematic for our analysis because we designed this stimulus to detect inattentive participants.

Sound generation.

First, a template MIDI file was created using manual entries. The template comprised 36 pairs of snare taps at equal intervals (interbeat interval = 500 ms) by two players in perfect synchronization. The timing of these taps was replaced by the tap timing series generated using the aforementioned procedure with a MATLAB MIDI toolbox [58]. The MIDI files were subsequently converted into WAV files using FluidSynth (https://www.fluidsynth.org/), part by part. The output WAV files were combined and re-exported as a single stereo WAV file using MATLAB. Different sound fonts were used for Player 1 (FluidR3_GM, https://github.com/urish/cinto/tree/master) and Player 2 (GeneralUserGS, https://schristiancollins.com/generaluser.php) to prevent unnatural sound localization. Additionally, the left channel of the WAV file contained waveforms synthesized from Players 1 and 2 with an amplitude ratio of 4:1, whereas the right channel contained waveforms synthesized with the opposite amplitude ratio.

Participant recruitment

The participants in the online listening experiments were recruited through a crowdsourcing platform (CrowdWorks, Inc.). They applied to three recruitment announcements posted on the platform and participated in one or more of the experiments. Each announcement corresponded to a distinct experimental condition defined by varying the model parameters of the stimuli. The recruitment period for all announcements was from 05/03/2024–10/03/2024. For each recruitment announcement, 150 participants participated in the experiment, with some participating in more than one experiment. Participants who participated in multiple experiments were identified, and all the participants were assigned unique IDs.

This study was approved by the Ethics Committee of the Graduate School of Human Development and Environment of Kobe University (approval number: 697). The participants were provided a detailed explanation of the research and written informed consent was obtained from all the participants, who clicked on a checkbox to approve before starting the tasks on the website for the experiment.

Music sophistication questionnaire

We used a musical sophistication questionnaire, the Goldsmith Musical Sophistication Index (G-MSI), in the Japanese language [59,60] to assess the relationship between participants’ abilities for music perception or the performance and perception of the statistical structure of timing sequences. This index is a widely used measure that has been translated into various languages since its release in 2014; it has demonstrated good psychometric properties and correlation with performance on listening tests that measure two different abilities: melodic memory and beat perception [59,60]. We calculated these scores from the responses according to a previous report [60]. However, methods for measuring perceptual abilities related to the ensemble are currently limited to beat alignment tests [61–63], that is, tests related to the judgment and realization of synchrony between beats (or beats and actions). Thus, in the present study, we also asked participants about their experiences of habitual participation in ensembles (yes/no selection and, if yes, genre selection) for an exploratory analysis.

Experimental procedure

Participants registered for the experiment on the crowdsourcing platform and received a URL directing them to the experiment website created using lab.js [64]. Initially, the screen displayed the experiment description and a checkbox for providing informed consent. Upon agreeing and proceeding, the participants were asked questions regarding their age, gender, and audio playback environment.

Next, the participants adjusted the sound volume according to the on-screen instructions. They first set the volume to 25% on their computer and adjusted it to a comfortable level while listening to the same drum-tapping sounds used in the experiment. Subsequently, a headphone screening test was conducted to confirm whether the participants used headphones or earphones [65]. The participants listened to three tones and selected the weakest one. The three tones were 200-Hz pure tones but with (i) diotic, (ii) diotic and 6-dB softer, and (iii) dichotic antiphase presentations. This test was repeated six times, and only participants who answered correctly at least five times were allowed to proceed to the subsequent stages.

An explanation of the experimental task was then provided, which included an example of an auditory stimulus and the following instructions: “The sound you just heard is a computer-generated replica of two players playing a snare drum together.” “In the experiment, you will be asked to judge whether the fluctuations in the timing of the drum sound resemble those of a human playing or not.” Following these instructions, HUM stimuli were presented as an example of “human-like” fluctuations, while RAN and ISO stimuli were presented as examples of “not human-like” fluctuations, twice, twice, and once, respectively.

Subsequently, the participants completed the practice trials. During the practice trials, the participants were instructed to listen to a stimulus and click the “human-like” button if they perceived it as HUM, and “not human-like” button if they perceived it as RAN or ISO within 2.5 s. The practice comprised nine trials, with HUM, RAN, and ISO presented three times each in a random order. The five HUM–RAN pairs used in the examples and practice were selected to represent the range of the cross-correlation structures of the HUM presented in the main task. These stimuli corresponded to the 1^st^, 5^th^, 10^th^, 15^th^, and 20^th^ ranks when the 20 stimuli were sorted by their distance from the average cross-correlation structure (calculated as the sum of the squared difference from the ensemble mean of the WDCC function) output from the model simulations. The participants were instructed to keep their eyes on a gaze point at the center of the monitor while listening to the stimuli. After the button response, they received feedback on whether their answers were correct or incorrect during the practice trials. Regardless of the correct response rate in the practice trials, the participants proceeded to the main experiment without being screened after completing one set of practice trials. This was because the extent to which HUM and RAN could be discriminated at the beginning was unclear.

The main experiment comprising three blocks of 12 trials was then conducted. In each block, five HUM and RAN stimulus trials and two ISO stimulus trials were presented in a random order. The participants were asked to judge whether the sound they heard was HUM or otherwise (if the former, they selected “human-like;” if the latter, they selected “not human-like”) within 2.5 s. We employed the two-alternative forced-choice paradigm to detect subtle differences in listeners’ perceptions and obtain perceptual bias and sensitivity according to signal detection theory. The participants were instructed to keep their eyes on a gaze point at the center of the monitor while listening to the stimuli. They were allowed to take breaks of any length between the blocks. After completing all trials, the participants were asked to respond to all the questions in the Japanese version of the G-MSI [59,60]. They were also asked to specify whether they had ever engaged in regular group musical activities, the genre of music they had experienced, and the duration of their involvement in years. The web system provided a completion code to each participant after they completed these questionnaires.

Data screening

In online experiments, participants can participate at any location and time, making it impossible for the experimenter to monitor their behavior. This raises concerns about dishonest or careless response behaviors (satisficing) [66,67]. To ensure that the analyses were performed only for participants who followed the instructions, data were excluded if any of the following criteria were met: the participant reported recognizing noise sources other than their computer, provided an answer other than the name or model number of headphones or earphones in the audio playback environment question, ran out of time in the main task twice or more, or judged ISO as “human-like” twice or more (The ISO is easily discriminable, and it is only presented six times during the test phase. In fact, the number of errors of judging the ISO as “human-like” was once or less for more than 90% of the participants: see Table S3 in S1 Appendix A). The numbers of participants who passed the screening were 100, 100, and 87 for b = 1.5, 1.0, and 0.5, respectively. The demographics are presented in Table 2 (see Table S4 in S1 Appendix A for the music genres of the participants who passed the screening). Participants who participated in multiple b conditions were identified using their user ID on the crowdsourcing platform. The number of unique participants who passed the screening was 181:34 participants participated in all conditions; 53 participants participated in b = 1.5 and b = 1.0 conditions; 47 participants participated in b = 1.5 and b = 0.5 conditions; 40 participants participated in b = 1.0 and b = 0.5 conditions; 34, 41, and 34 participants participated in only b = 1.5, b = 1.0, and b = 0.5 conditions, respectively.

Table 2: Demographics of participants who passed the screening.

Statistical analyses

The analyses were conducted in an exploratory manner, focusing on whether the proportion of “human-like” responses, along with sensitivity and bias of discrimination (d’ and C, respectively, in signal detection theory [68]), varied depending on the stimulus, parameter b, and participant profile. The d’ and C are derived from the proportion of “human-like” responses to HUM and RAN. A larger positive d’ indicates a higher percentage of “human-like” responses to HUM than to RAN, whereas a larger negative d’ indicates the opposite. A larger positive C suggests a bias towards “human-like” responses, whereas a larger negative C indicates the reverse. The d’ and C were calculated using MATLAB R2023a (Mathworks, USA). The S4 Dataset contains data used for statistical analyses, including data removed during screening.

We adopted a linear mixed model (LMM: sum contrast coding) in all the statistical analyses because fixed effects in the LMM have been empirically demonstrated to be robust to violations such as homogeneity of variance, normality of residuals, and failure to estimate random effects [69,70]. In our data, all random effects could not be estimated and Levene’s test and Kolmogorov–Smirnov test suggested violations of the homogeneity of variance and normality of residuals in some parts of the analyses (see Results). Thus, we adopted LMM analyses and, for reference, showed the results using robust standard errors (Table S5-S9 in S1 Appendix A). LMM analyses were performed using the lme4 [71] and lmerTest packages [72] in R4.2.2 (R Core Team). All the LMM models were fitted with a restricted maximum likelihood (REML), and significance was calculated using Satterthwaite’s method for estimating the degrees of freedom and p-values. The significance level was set at p < .05. When analysis of variance (ANOVA) detected effects above borderline significance (p < .10), post hoc power analyses were performed using the simr package [73], and a 95% confidence interval was reported (number of simulations: 1000). Additionally, the effect sizes (standardized fixed effect coefficients and inclusive R^2^ [74]) and robust standard errors for fixed effect estimates were calculated using the effectsize [75], partR2 [73] (number of bootstraps: 1000) and clubSandwich [76] packages.

As demonstrated in S2 Appendix B, the correlations between the G-MSI score and the accurate response rate in practice trials, d’, or C were weak and inconsistent. Meanwhile, participants with ensemble experience generally exhibited a higher total G-MSI scores than those without such experience (Table 2). A linear mixed model analysis was performed to determine whether the total G-MSI scores differed across ensemble experiences (yes and no) and levels of parameter b (1.5, 1.0, and 0.5). The lme4 model formula was G-MSI score ~ b * ensemble experience + (1 | participant ID). As discussed in the Results section, the fixed effect of ensemble experience was significant, prompting subsequent analyses focused on the influence of ensemble experience.

We compared the proportion of “human-like” responses to HUM and RAN (the number of “human-like” responses divided by the number of responses that did not run out of time) using LMM to assess the extent to which HUM and RAN were discriminated. The fixed-effect variables included stimulus type (HUM and RAN), parameter b (0.5, 1.0, and 1.5), and ensemble experience (yes or no). Participant ID was included as a random effect variable. The lme4 model formula was the proportion of “human-like” response ~ stimulus type * b * ensemble experience + (1 | participant ID). ISO was excluded from the analysis because it was specifically designed to be easily distinguished to identify satisficing and inattentive participants. Furthermore, d’ and C were analyzed using LMM, with b and ensemble experience as fixed effect variables and participant ID as a random effect variable. The lme4 model formulas were d’ ~ b * ensemble experience + (1 | participant ID) and C ~ b * ensemble experience + (1 | participant ID). In addition, to examine the effects of ensemble experience on the practice trials, the accurate response rate in the practice trials was analyzed using LMM, with parameter b and ensemble experience as fixed effect variables and participant ID as a random effect variable. The lme4 model formula was accurate response rate ~ b * ensemble experience + (1 | participant ID).

Results

Ensemble experience and G-MSI scores

To examine whether the total G-MSI score varied across experimental conditions or ensemble experience, an LMM analysis was conducted with the total G-MSI score as the dependent variable, and b and ensemble experience as fixed effects variables. The model converged (REML criterion = 2418.5; marginal R^2^ [95% CI] =.176 [0.082, 0.258]). The Levene’s test did not suggest a significant violation of the assumption of homogeneity of variances: F(5, 281) = 0.497, p = .778). The Kolmogorov–Smirnov test suggested a significant violation of the normality of the residuals: D = 0.155, p < .001. The ANOVA revealed a significant effect of ensemble experience (F(1.00, 180.24) = 39.42, p < .001, $[eqn]$  = .179, 95% CI of simulated power = 99.63–100.0%), suggesting that participants with ensemble experience had significantly higher total G-MSI scores than those without it. The main effect of b and interaction between b and ensemble experience were not significant (F(2.00, 107.42) = 2.15, p = .121, $[eqn]$  = .039; and F(2.00, 107.42) = 0.14, p < .866, $[eqn]$  = .003, respectively).

The fixed effect estimates are presented in Table 3 (standardized and robust estimates and inclusive R^2^ are presented in Table S5 in S1 Appendix A). In addition to the effects of the ensemble experience, participants in the b = 0.5 condition exhibited significantly higher total G-MSI scores than those in the other conditions (t(107.51) = 2.07, p = .040). The other fixed effects are not significant (ps > .05).

Table 3: Fixed effect estimates on total G-MSI score.

Responses to the HUM and RAN stimuli

Fig 2 presents the percentage of “human-like” responses for HUM and RAN, alongside the chance level (50%, as participants chose between “human-like” and “not human-like” in all trials). To examine whether HUM obtained more “human-like” responses than RAN, we performed a LMM analysis with the “human-like” response rate as the dependent variable and stimulus type, b, and ensemble experience as independent variables. The model converged (REML criterion: 4609.9). The Levene’s test did not suggest a significant violation of the assumption of homogeneity of variances: F(11, 562) = 1.60, p = .094. The Kolmogorov–Smirnov test did not suggest a significant violation of the normality of the residuals: D = 0.02, p = .913. The ANOVA revealed a significant effect of stimulus type (F(1.00, 384.73) = 5.50, p = .020, $[eqn]$  = .014, 95% CI of simulated power = 38.92–45.13%) and borderline interaction of stimulus type and ensemble experience (F(1.00, 384.73) = 3.79, p = .052, $[eqn]$  = .010, 95% CI of simulated power = 47.15–53.44%). Other main effects and interactions were not significant (b: F(2.00, 554.60) = 0.74, p = .476, $[eqn]$  = .003; ensemble experience: F(1.00, 172.17) = 1.88, p = .172, $[eqn]$  = .011; stimulus type × b: F(1.00, 384.73) = 3.79, p = .052, $[eqn]$  = .010; b × ensemble experience: F(2.00, 554.60) = 0.38, p = .683, $[eqn]$  = .001; stimulus type × b × ensemble experience: F(2.00, 384.73) = 1.07, p = .344, $[eqn]$  = .006). These findings suggest that participants with ensemble experience were more likely to perceive HUM as “human-like” than RAN compared to participants without ensemble experience.

Comparison of the percentage of “human-like” responses by group and stimulus.chance levels Panels A and B display the data for participants with ensemble experience, whereas panels C and D show the data for participants without ensemble experience. (A) and (C): box charts illustrating the percentages of participants’ responses as “human-like” to the RAN (blue) and HUM (red) stimuli, with adjacent dots representing individual participant data. (B) and (D): box chart for the difference between the percentage of “human-like” responses to RAN and HUM, with neighboring dots indicating each participant’s data. The dashed lines indicate chance levels (50% for panels A and B, and 0 for C and D).

The fixed effects estimates are shown in Table 4 (standardized and robust estimates and inclusive R^2^ are presented in Table S6 in S1 Appendix A). HUM obtained significantly more “human-like” responses overall (t(384.73) = 2.35, p = .020). Furthermore, a borderline interaction between stimulus type and ensemble experience was revealed: participants without ensemble experience provided fewer “human-like” responses to HUM (t(384.73) = –1.95, p = .052). The other fixed effects were not significant (ps > .10).

Table 4: Fixed effect estimates on “human-like” response rate.

Ensemble experience, sensitivity, and response bias

To assess whether the group differences in the response tendencies observed in the above analysis stemmed from sensitivity or response bias, LMM analyses were conducted on d’ and C as dependent variables, with b and ensemble experience as fixed effects variables. Fig 3 illustrates the distributions of d’ and C. Both models converged (REML criterion: −740.1 for d’ and −1002.1 for C). The Levene’s test did not suggest a significant violation of the assumption of homogeneity of variances: F(5, 281) = 0.74, p = .591 for d’, and F(5, 281) = 1.44, p = .201 for C. The Kolmogorov–Smirnov test did not suggest a significant violation of the normality of residuals: D = 0.05, p = .445 for d’ and D = 0.06, p = .219 for C. The ANOVA on d’ revealed the borderline main effect of ensemble experience (F(1.00,281.00) = 3.39, p = .066, $[eqn]$  = .012, 95% CI of simulated power = 40.30–46.54%). The main effect of b and interaction between b and ensemble experience were not significant (b: F(2.00,281.00) = 1.03, p = .357, $[eqn]$  = .007; b × ensemble experience: F(2.00,281.00) = 1.00, p = .371, $[eqn]$  = .007). Conversely, the ANOVA on C revealed that all the main effects and interactions were not significant (b: F(2.00, 162.90) = 0.93, p = .396, $[eqn]$  = .011; ensemble experience: F(1.00, 171.75) = 2.05, p = .154, $[eqn]$  = .012; b × ensemble experience: F(2.00, 162.90) = 0.38, p = .687, $[eqn]$  = .005). Thus, although the evidence remains limited, group differences in response tendencies were more likely attributable to sensitivity rather than to response bias.

Comparison of sensitivity and response bias.Panels A and B present the data for participants with ensemble experience, whereas panels C and D present the data for those without ensemble experience. (A) and (C): box charts for sensitivity d’, with adjacent dots representing individual participant data. (B) and (D): box charts for bias C, with neighboring dots indicating each participant’s data.

The fixed effect estimates for d’ are provided in Table 5 (standardized and robust estimates and inclusive R^2^ are presented in Table S7 in S1 Appendix A). Participants without ensemble experience demonstrated a slightly lower sensitivity than those with ensemble experience (t(281) = –1.84, p = .066). The fixed effect estimates for C are shown in Table 6 (standardized and robust estimates and inclusive R^2^ are presented in Table S8 in S1 Appendix A). No significant effects were observed across any of the terms (ps > .05).

Table 5: Fixed effect estimates of sensitivity d’.

Table 6: Fixed effect estimates on bias C.

Accurate response rate in practice trials

The effects of b and ensemble experience on the accurate response rate in the practice trials were analyzed using an LMM to check whether the responses of the participants with ensemble experience were accurate in the practice trials. The model converged (REML criterion: −275.80). The Levene’s test suggested a significant violation of the assumption of homogeneity of variances: F(5, 281) = 2.61, p = .025. The Kolmogorov–Smirnov test suggested a significant violation of the normality of the residuals: D = 0.12, p < .001. The ANOVA revealed that all the main effects and interactions were not significant (b: F(2.00, 281.00) = 1.16, p = .316, $[eqn]$  = .008; ensemble experience: F(1.00, 281.00) = 0.85, p = .357, $[eqn]$  = .003; b × ensemble experience: F(2.00, 281.00) = 0.96, p = .386, $[eqn]$  = .007), suggesting that participants with ensemble experience were not necessarily able to discriminate better from the practice stage, while the minimum accurate response rate appeared to be slightly high (Fig 4).

Comparison of accurate response rate in practice trials.Panels A and B present data for participants with and without ensemble experience, respectively, with adjacent dots representing individual participant data.

The fixed effect estimates are provided in Table 7 (standardized and robust estimates and inclusive R^2^ are presented in Table S9 in S1 Appendix A). None of the fixed effects was significant (ps > .05).

Table 7: Fixed effect estimates of the accurate response rate in practice trials.

Discussion

This study aimed to examine whether it is possible to discriminate between paired ITI series generated by the model of Okano et al. (2019) (HUM) and paired randomized ITI series (RAN) and whether there is a relationship between discrimination ability and musical background. Overall, HUM elicited more “human-like” responses than RAN. In addition, although the statistical effect was borderline and the statistical power was limited, the results suggested that participants without ensemble experience had difficulty discriminating, whereas several participants with ensemble experience could discriminate to some extent. In addition, musical sophistication did not correlate with sensitivity or judgment bias.

The observed perceptual difference between HUM and RAN would reflect the difference in their cross-correlation structures. Previous studies on micro-timing have demonstrated that dynamical properties of timing can be distinguished despite the asynchrony and ITI differences contained in micro-timing being close to the perceptual threshold. The difference in the responses to HUM and RAN in the present study extends the scope of the findings of the previous studies to the level of coordination structures between partners. HUM were stimuli with mutual timing adaptation structures, whereas RAN were stimuli without consistent cross-correlation structures. Although the evidence is limited, the fact that these can be distinguished to a certain degree is a necessary requirement for the question of whether there is individuality and preference in the fluctuation of the timing and coordination of partners in an ensemble.

Differences in b in the HUM are linked to the strength of the mutual timing adaptation [30,31,57,55]; the larger b, the more the mutual timing adaptation pattern is strengthened. This initially led us to hypothesize that a larger b would make discriminating between HUM and RAN easier. However, the relationship between the magnitude of b and the rates of judgment, sensitivity, and bias was unclear. While a relatively small dependence between partners was sufficient for participants with ensemble experience to discriminate, a greater dependence did not necessarily facilitate discrimination within the scope of this study. Rather, although the group differences in sensitivity d’ were not statistically significant, they tended to be more pronounced in conditions with smaller b. This may be because RAN was judged to be more “human-like” on average than HUM, with b = 0.5, although this difference was not statistically significant in the group without ensemble experience. For these participants, b = 0.5 may be too small a dependence for judgment. The experimental design, in which b was an inter-subject factor, weakened the statistical power and made it difficult to draw definitive conclusions. Setting b as an intra-subject factor and more pronounced b values will contribute to validating these points.

This study suggests a possible connection between ensemble experience and the ability to discriminate between fluctuations and coordination in multipart auditory sequences. If so, participants with ensemble experience may have learned the difference between HUM and RAN through several practice sessions with feedback, or may have already had the potential for this. Although the participants with ensemble experience did not exhibit a significantly higher rate of accurate responses in the practice trials, they demonstrated a slightly higher minimum accurate response rate during the practice trials and slightly greater sensitivity during the main experiment. These results motivate further validation studies with a more precise grouping of ensemble experiences.

From a perceptual information processing perspective, our model captures mutual timing adaptation, which is a part of the cognitive-motor skills involved in ensemble performance. The cross-correlation structure of HUM stimuli is the outcome of a player’s mutual adaptation [48]. From this perspective, an ensemble experience may cultivate an understanding of the fluctuations driven by mutual timing adaptations among individual performers. Conversely, within the dynamical systems framework, ensemble coordination is interpreted as an emergent collective behavior arising from multiple coupled oscillators (performers) rather than resulting from short-term, pairwise interactions between individuals [76]. From this perspective, ensemble experience may nurture an understanding of fluctuations as a group-level collective behavior of performers, rather than individual adaptations.

However, it should be noted that mutual timing adaptation does not necessarily equate to high-quality coordination. For instance, Repp (2010) found that, in a sensorimotor synchronization experiment, participants with extensive musical experience showed slower phase correction responses, suggesting that faster phase correction is not always better for performance [77]. Further research is needed to explore the specific mechanisms or perspectives that enable discrimination between structured and random fluctuations, and how ensemble experience contributes to this ability. Furthermore, the “human-like” or “not human-like” labels used for judgment in this study were only operational. Therefore, future research should explore task instructions that more directly reflect the concept of “coordination.”

Scientific research on micro-timing in isochronous music ensembles remains in its early stages, necessitating investigations in minimal settings. Thus, this study focused on the duo, the smallest functional unit of ensembles, to examine the effects of the structures of timing fluctuation and coordination. Although the discriminability levels obtained in this study were at the borderline of statistical significance, further verification is needed. The results offer valuable foundational data for future research exploring additional factors. A future direction for ensemble micro-timing research is to extend the investigation to non-temporal dimensions. For example, a study demonstrated that a misalignment in the micro-timing of snare drums is rated worse than a similar misalignment in bass drums [77]. Human responses to timing errors are stronger for high-frequency tones than low-frequency tones [78]. Timbre affects perceptual note attacks and their onset [79]. How cognitive and biomechanical constraints, visual cues, and social interactions modulate the coordination structure of fluctuations remains unclear. Therefore, research using stimuli with higher ecological validity that considers these factors is necessary. In such cases, using a model based on the Kuramoto model may be effective [49,80]. These formulate temporal adaptation based on the overall behavior of the system rather than on pairwise phase and period corrections; thus, they may be more suitable than our model when dealing with situations involving three or more people. The two-alternative forced-choice task adopted for the online experiments may have impaired the resolution of stimulus discrimination. This point is also worth verifying experimentally using confidence rating scales. Despite the above limitations, this study highlights the potential importance of micro-timing in acoustic communications and offers a valuable framework for examining the discriminability of the correlation structures of paired micro-timing fluctuations.

The coupled oscillator model used for stimulus generation is expressed in a form that facilitates comparison with previously established models of discrete phase and period corrections [81–83]. This formulation has the advantage of allowing the easier manipulation of the cross-correlation structure compared to the formulation based on the Kuramoto model [47,49,84]. Additionally, its formulation based on the continuous time evolution of phases enables the simulation of joint rushing, that is, tempo acceleration in an ensemble situation [32,57,85–89], which cannot be replicated by a previous discrete-time linear phase-period correction model [81–83,90], and offers superior properties of ease of expansion to larger ensemble contexts.

It is also noteworthy that the ability to discriminate between stimuli did not correlate with any (sub)scales of the Gold-MSI scale. As the current Gold-MSI lacks a subscale specifically addressing ensemble ability, we directly inquired about the participants’ ensemble experience as a precautionary measure. The results revealed a possible link between the ensemble experience and discrimination ability. Unfortunately, our survey did not clarify how much or what type of ensemble experience enhanced the discrimination ability (participants’ genre backgrounds are summarized in S1 Appendix A Table S1 in the Supporting Information). This limitation arose because we initially hypothesized a correlation between the G-MSI score and d’, and to accommodate the online experiment situation, we prioritized minimizing the survey duration. Consequently, the ensemble experience inquiries were restricted to the minimum required for an exploratory analysis. Further research is needed to determine which aspects of the ensemble experience enhance the ability to discriminate between the structures of fluctuation and coordination.

This study quantitatively demonstrated through a systematic listening experiment that discriminability in the timing structures of fluctuations and coordination exists not only in solo classic performances but also in ensembles (at least duos) of isochronous sequences, and that listeners with certain qualities may be able to discriminate between these differences. Although unsolved issues remain, such as the nature of micro-timing and the substantive “humanness” of fluctuation and coordination, the most crucial significance of this study lies in its demonstration that these concepts are not mere illusions perceived by enthusiasts.

Supporting information

S1 Appendix ASupplementary tables. Tables showing the results of supplementary analyses.(DOCX)

S2 Appendix BSupplementary figures. Scatter plots and correlation matrices for correlations between sensitivity, bias, and G-MSI scores.(DOCX)

S3 FileMP3 files of stimuli.(ZIP)

S4 DatasetData for statistical analyses.(CSV)