Intermittent control and retinal optic flow when maintaining a curvilinear path

TL;DR

This study explores how humans use visual cues to navigate curved paths, finding that they make intermittent corrections based on retinal optic flow and vehicle heading.

Contribution

The paper introduces a novel method using particle swarm optimization to analyze intermittent steering corrections and measures response times for optic flow and heading-based cues.

Findings

Human response time for retinal optic flow-based cues is approximately 0.14 seconds.

Response time increases to 0.44 seconds when using heading-based cues.

Removing the visible steering wheel adds an additional 0.17 seconds delay in response time.

Abstract

The topic of how humans navigate using vision has been studied for decades. Research has identified the emergent patterns of retinal optic flow from gaze behavior may play an essential role in human curvilinear locomotion. However, the link towards control has been poorly understood. Lately, it has been shown that human locomotor behavior is corrective, formed from intermittent decisions and responses. A simulated virtual reality experiment was conducted where fourteen participants drove through a texture-rich simplistic road environment with left and right curve bends. The goal was to investigate how human intermittent lateral control can be associated with the retinal optic flow-based cues and vehicular heading as sources of information. This work reconstructs dense retinal optic flow using a numerical estimation of optic flow with measured gaze behavior. By combining retinal optic flow with the drivable lane surface, a cross-correlational relation to intermittent steering behavior could be observed. In addition, a novel method of identifying constituent ballistic correction using particle swarm optimization was demonstrated to analyze the incremental correction-based behavior. Through time delay analysis, our results show a human response time of approximately 0.14 s for retinal optic flow-based cues and 0.44 s for heading-based cues, measured from stimulus onset to steering correction onset. These response times were further delayed by 0.17 s when the vehicle-fixed steering wheel was visibly removed. In contrast to classical continuous control strategies, our findings support and argue for the intermittency property in human neuromuscular control of muscle synergies, through the principle of satisficing behavior: to only actuate when there is a perceived need for it. This is aligned with the human sustained sensorimotor model, which uses readily available information and internal models to produce informed responses through evidence accumulation to initiate appropriate ballistic correction, even amidst another correction.

Figures

Retinal optic flow and retinotopic flow. The illustration demonstrates how visual motion flow is projected to the retina in the eye, creating the retinal optic flow or retinotopic flow sensation the observer experiences. The retinal optic flow field accounts for the rotational eye movements exhibited during gaze fixation and smooth pursuit, achieving stabilization of the retinal image. The main difference between retinal optic flow and retinotopic flow is the stereographic projection, i.e. projection to the sphere skewing coordinate system.Fig. 2Visual input, image processing, and data visuali

Retinal optic flow and retinotopic flow. The illustration demonstrates how visual motion flow is projected to the retina in the eye, creating the retinal optic flow or retinotopic flow sensation the observer experiences. The retinal optic flow field accounts for the rotational eye movements exhibited during gaze fixation and smooth pursuit, achieving stabilization of the retinal image. The main difference between retinal optic flow and retinotopic flow is the stereographic projection, i.e. projection to the sphere skewing coordinate system.Fig. 2Visual input, image processing, and data visuali

Full text

Introduction

Vision is arguably one of the most complex perception senses enabling animals to navigate and exploit the ever-dynamic surrounding world as we understand it^1–3^. Although there is a scientific consensus that vision perception plays a critical role, it is not yet fully understood how humans use it to guide, propel, or control their egomotion on foot^4,5^ or in vehicles. It has been proposed that visual flow, more specifically the retinal optic flow, is exploited guiding the planar curvilinear locomotion in a textured environment^6–13^, e.g. riding a bicycle or driving a road vehicle. It has often been argued that task-driven gaze fixation transforms and stabilizes retinal optic flow patterns to counteract the chaotic relative motion that may occur during locomotion^11,14^ and further due to anticipation of trajectory planning^4,15–17^. Our work integrates various key findings across scientific domains and demonstrates how visual stimuli, such as retinal optic flow, may result in an intermittent ballistic correction in human locomotor control. This work offers insight into how visual input may relate to action output response for correcting egomotion.

It is an ongoing discussion whether human locomotor behavior can be solely described through a strong online approach, a strong model-based approach, or as a hybrid composite of both^18^. A strong online control mechanism is characterized by solely relying on the immediate available external information to act upon. In contrast, a strong model-based approach uses an internalized model to process the external information, transform it, update one’s belief, or predict the internal states, which are then used to act upon. Recently, there has been increasing evidence that humans must have internalized models to explain observed predictive behavior in humans^17,19,20^ (model-based control). Simultaneously, other research implies that initial muscle motor control processes are completed during the motor planning phase in the human brain^21–25^, suggesting that humans complete the initial action response before any other auxiliary higher cognitive processes as prediction and planning are fully completed (online control).

Intermittent control as a phenomenon is commonly found in psychology and biomechanical contexts. This biomechanical control emerges from the fact that muscles are only capable of generating force through contraction, often around a joint. Thus biological systems are evolutionarily optimized for energy efficiency, co-contraction, muscle synergy, etc. Since energy is a valuable resource, things ought to change only when there is a sufficient need for it. Despite intermittent control being an established topic in experimental psychology^26,27^, it is only recently gaining attraction outside of its field when studying human control behavior^28,29^. In contrast to optimal control and classical control theory^30^, human control behavior should be understood as an intermittent control system where corrections are intermittently applied via trains of ballistic movements to achieve a task^21,27^. This has been successfully developed, applied, and demonstrated through computational model frameworks^28,29,31^. It has been found that the human-operated steering wheel for lateral control in vehicles also exhibits the same intermittency property^28^.

In neuromuscular modeling using the minimum-jerk model^32^, these ballistic correction movements correspond to so-called reaching movements, rapid aimed limb movement from point to point, with characteristics of sigmoid positional and bell-shaped velocity profiles^33,34^. To achieve one single ballistic reaching movement, a set of building blocks of neural and muscle networks, or muscle synergies, are activated and recruited^35–38^. It is believed that the central nervous system uses these muscle synergies to simplify the coordination and control of complex movements to alleviate the cognitive load. This is done by mapping initial states and task-specific goals to time-varying activation of muscle synergies on a higher level, contrary to individually micro-controlling each muscle primitive in a centralized manner^35,36^.

Various research disciplines have contributed significantly to our understanding of human locomotor control behavior. However, these contributions are often made in isolation in their respective research domain without any further integration into other fields. There have been limited attempts to incorporate these individual findings into a more unified and harmonized understanding of human locomotor control. Unification of these ideas paints the larger picture and extrapolates their findings outside of their place of inception and driving science forward^39,40^. In this work, we investigate and report the findings observed in human driving behavior in curvilinear paths in a visually textured environment, bridging the gaps across the research fields. In particular, our objective is to study in detail how retinal optic flow can serve as a perception cue for intermittent control, e.g. how the perception of visual motion may trigger a ballistic locomotor correction response. Emerging from this, the following research questions are posed:

1. How well can intermittent control describe human steering behavior by quantifying overlapping ballistic steering corrections?
2. What are the human response times, from visual stimuli (retinal optic flow and heading) onset to steering correction onset?
3. How does the human locomotor behavior change when removing vehicle-fixed objects visually?
4. How can visual perception cues trigger or correlate with ballistic correction as reaching?

In order to investigate the posed research questions, our work identifies and deals with three significant hurdles of quantification: retinal optic flow, ballistic adjustments in human intermittent control, and human mental processing latency. In the first part, the retinal optic flow field as perceived by the locomoting agent can be qualitatively reconstructed using a set of idealized assumptions of smooth pursuit and experimental data. In the second part, ballistic adjustment is derived based on the theories of intermittent control and demonstrated on the one-dimensional steering wheel angle in our work. In addition, our work presents a novel experimental method using a stochastic approach to identify each ballistic correction in experimental data with complex overlapping ballistic corrections. Finally, these results allow for an estimation of human cognitive latency using cross-correlation analysis of perceived stimuli to a correctional action for lateral control. In addition, locomotor performance is compared by offering a fully unobstructed view of the scene by making the steering wheel invisible. Doing so may provide insight into how the perception of the ego-state impacts human locomotor control performance.

Background

Human perception in curvilinear path

For decades, researchers have long debated the intricate details of what and how humans perceive the environment for navigation^9,41,42^. One such debate is whether retinal optic flow, without recovering the heading or exploiting extra-retinal information, may be used in curvilinear locomotor control^6,7,12,42,43^. Although it has been shown mathematically^44^ that it is possible to make the correct steering judgments by relying solely on the immediate available retinal optic flow, it has yet to be decisively concluded to be the case. It is still poorly understood if these steering judgments are informed solely on retinal optic flow, combined with extra-retinal information, or allocentric cues alone. One contributing factor to this problem is the lack of relevant high-quality data to complete the debate. This has effectively resulted in the status quo in research regarding human sensorimotor models incorporating gazing until recently with the rapid development of modern eye-tracking systems.

Meanwhile, visual flow has been widely adopted across scientific and engineering domains. This has created ambiguity as researchers across domains have used the terms optic flow and retinal optic flow (or retinal flow in early literature) interchangeably. In the literature, optic flow is often considered in terms of the locomotor axis or the velocity vector of the observer. In this work, the optic flow will be computed and referred to the visual flow fields fixed with respect to the head-fixed coordinate system (due to the virtual reality–head-mounted display). The retinal optic flow^10,11^ here will refer to the emergent patterns of visual motions that would be projected to the human eye but omitting the stereographic projection (projection to a sphere). This mainly accounts for expressed eye movements. If the final stereographic projection to the retina is performed, this could be referred to as the retinotopic flow fields or visual flow in retinotopic coordinates in the neurocognitive science. This can be imagined with a fixed coordinate system to the eye as the visual motion flow is projected onto the retina, see Fig. 1. It is foremost the retinotopic flow field stimulus that the observer experiences. However, in our work, the retinal optic flow is mainly considered without loss of generality when the stereographic projection is omitted.

Researchers advocating for retinal optic flow as the main perceptual cue for curvilinear locomotor control have proposed a simplified perception-control heuristic^6,7,9^. The proposal goes in steps: (a) Fixate the gaze close to the intended path, (b) the directions of the retinal optic flow over the intended path will then indicate understeering or oversteering, and finally (c) if the retinal optic flow directions of the intended path are pointing downward, then the future path and intended path coincide and no further correction is needed. This behavioral heuristic can simply be formulated as “you look where you are going” and “you go where you are looking”. Interestingly, the gazing behavior implied by the stated heuristic may be a conditioned behavior as taught by advanced driving instructors and possibly exhibited by licensed (experienced) drivers^7^. There is a significant demand for proper steering maintenance during critical moments, such as curve bends, resulting in a larger cognitive load^15^. This may suggest or motivate why such a strategy is deployed to cope with this.

From a research point of view, the challenge lies in quantifying the retinal optic flow of the intended path of the agent as stated by the simplified heuristic if such a quantity exists. Given the task that the agent is supposed to maintain their current future path so that this path lies within their designated road lane, then the designated road lane should contain their internalized projected intended path. This still holds when the agent cuts corners when locomoting in a curve bend. Then the intended path is shifted closer to the corner side of the lane but still within the boundaries of the designated road lane. For this work, it will be assumed that the designated road lane visually contains the intended path of the agent as the task is to stay in the lane while completing the task. To then construct a quantity as described by the heuristic, the spatial circular mean angle of retinal optic flow over the intended path becomes

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{\theta }_{{{\text{rof}}}} = \arg \left( {\sum\limits_{{\theta _{{{\text{rof}}}} \in \Omega _{{{\text{lane}}}} }} {\exp } \left( {i\theta _{{{\text{rof}}}} } \right)} \right),$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _\text {rof}$$\end{document} is the directional angle of the dense retinal optic flow field (see the visualization of retinal optic flow in Fig. 2c and further Fig. S1 in Supplementary Materials), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega _\text {lane}$$\end{document} is the set containing the dense visual flow of the visual lane (corresponds to the green area in Fig. 2b), and i is the imaginary unit. In this work, this quantity will be referred to as retinal optic flow angle instead of the spatial circular mean angle of the retinal optic flow over the intended path. Note that the naive arithmetic mean is not directly applicable to cyclic quantities, such as angles, due to the cyclic boundaries. The retinal optic flow angle can thus be understood as some extended computational quantity of the perceived retinal flow patterns of the intended path of the agent as described in the literature^6,7^. Here, the retinal optic flow angle is exclusively based on visual information, such as the retinal optic flow and lane perception without explicitly utilizing external information such as vehicular heading or vehicular lateral lane position.

For completeness, the locomotor heading cue will be expressed as the angle between the road curvature tangent to the vehicular heading as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \theta _\text{ h } = \theta _\text {road} - \theta _\text {veh}, \end{aligned}$$\end{document}

and vehicular lateral displacement as the lateral offset (lateral relative to the road curvature) from the middle line of the designated lane as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} y_\text{ l } = y_\text {lane} - y_\text {veh}. \end{aligned}$$\end{document}

The heading deviation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _\text{ h }$$\end{document} is closely related to lateral displacement velocity \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\dot{y}_\text {l}$$\end{document} . Disregarding the advanced vehicle and tire dynamics, the lateral displacement velocity may be simplified as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \dot{y}_\text {l} = ||\varvec{v}_\text {veh}||_2 \sin {\theta _\text{ h }} \approx ||\varvec{v}_\text {veh}||_2 \theta _\text{ h } \end{aligned}$$\end{document}

where the last approximation is valid for small angles of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _\text{ h }$$\end{document} .

Human motor control and intermittent control

To better understand if and when stimuli that trigger an expressed movement as a response, one needs to explore and understand the kinematics and characteristics of the expressed limb movements. When humans exert a voluntary rapid aimed limb movement in space, the movement is characterized using the minimum-jerk model^32^ of having a sigmoid positional curve as a distance to a target point^45^ and a bell-shaped velocity^33^ profile. This is also known as reaching movement in the fields of neuroscience^34,46,47^. In addition, observations made in experimental movement data reveal that velocity profiles of reaching often have a heavy tail^34^, originating from slower homing movement near the target point. However, for this work, the bell-shaped velocity profiles will be considered symmetric; see the Supplementary Material for further analysis and motivation. If the reaching can be considered symmetric, then one simple or ballistic reaching movement (ballistic correction) can be approximated as a symmetric Gaussian function

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\dot{\delta }}_\text{ b }(t,a_\text{ I }, \mu _\text{ I }, \sigma _\text{ I}) \triangleq a_\text{ I } \exp { \frac{-(t-\mu _\text{ I})^2}{2\sigma _\text{ I}^2}}, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a_\text{ I }$$\end{document} is the correction strength, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _\text{ I }$$\end{document} the mode of correction, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _\text{ I }$$\end{document} the correction rate parameter. The correction strength parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a_\text{ I }$$\end{document} may be physically interpreted as how strong the applied correction is and its direction, similar to an amplitude coefficient (indicated direction and length of the correction mode in Fig. S3 in Supplementary Materials). The correction mode parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _\text{ I }$$\end{document} determines the timeliness of correction, shifting the correction forward or backward in time (placement in time of the correction modes in Fig. S3 in Supplementary Materials). Lastly, the correction rate parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _\text{ I }$$\end{document} dictates how rapidly the correction is applied, smaller values shorten the duration and larger values prolong the duration.

In reality, however, the reaching movements are rarely cleanly executed by one correction, but by a highly complex train of corrections where they may overlap each other. The resulting complex movement of individual corrections can be described by the principle of superposition

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\dot{\delta }} (t) = \sum _{(a_\text{ I }, \mu _\text{ I }, \sigma _\text{ I}) \in \Omega _{{\dot{\delta }}}} \dot{\delta }_\text{ b }(t,a_\text{ I }, \mu _\text{ I }, \sigma _\text{ I}) \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega _{{\dot{\delta }}}$$\end{document} is a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3\times N_\delta$$\end{document} matrix

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Omega _{{\dot{\delta }}} =\begin{pmatrix} a_0 & a_1 & \dots & a_{N_\delta -1} \\ \mu _0 & \mu _1 & \dots & \mu _{N_\delta -1} \\ \sigma _0 & \sigma _1 & \dots & \sigma _{N_\delta -1} \\ \end{pmatrix}. \end{aligned}$$\end{document}

Each column in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega _{{\dot{\delta }}}$$\end{document} contains the parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(a_\text{ I }, \mu _\text{ I }, \sigma _\text{ I})$$\end{document} for a single ballistic correction. Refer to Supplementary Fig. S2 for illustrative examples of simple and overlapping complex reaching movement patterns.

Followed by the properties of the Gaussian function, the onset and offset of each intermittent correction movement can then be clearly defined as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} t_\text {I,\{on,off\}} \triangleq \mu _\text{ I } \mp 2\sigma _\text{ I } \end{aligned}$$\end{document}

by the given correction mode \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _\text{ I }$$\end{document} and correction rate parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _\text{ I }$$\end{document} . Following this convention, the total duration of the movement is defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$4\sigma _\text{ I }$$\end{document} which encompasses approximately 95.4 % of the complete theoretical movement.

The positional distance function is simply the primitive function of the velocity profile of the correction as defined in Eq. (5)

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \delta _\text{ b }(t,a_\text{ I }, \mu _\text{ I }, \sigma _\text{ I}) = \int a_\text{ I } \exp {\frac{-(t-\mu _\text{ I})^2}{2\sigma _\text{ I}^2}} \text{ d }t = a_\text{ I } \sigma _\text{ I }\sqrt{\pi /2} ~\text {erf}(\frac{t-\mu _\text{ I }}{\sqrt{2}\sigma _\text{ I }})+C \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {erf(\dots )}$$\end{document} is the Gauss error function and can not be expressed by elementary functions, and C is some constant determined by a known condition in position (usually a boundary condition). The full travel distance of the ballistic movement correction is then

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Delta \delta _\text{ b } (a_\text{ I }, \sigma _\text{ I}) = \int _{-\infty }^{\infty } \dot{\delta }_\text{ b }(t,a_\text{ I }, \mu _\text{ I }, \sigma _\text{ I}) \text{ d } t = a_\text{ I } \sigma _\text{ I } \sqrt{2\pi }. \end{aligned}$$\end{document}

The travel distance is fully determined by the correction strength and the speed parameter, which is to be expected. Examples of theoretical ballistic corrections for both cases of single correction and complex overlapping corrections are shown in Fig. S2 with the defined onset of offset timing.

A human-operated steering wheel shares the same fundamental characteristic as human reaching movements of a sigmoid positional curve and the bell-shaped velocity profile^28,29^. Under the assumption of normal steering handling and sufficiently small steering wheel angles to avoid hand shuffling, the ballistic correction duration parameter for the steering wheel was empirically found to be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _{{{\text{ I }}}} \in [0.0707{\kern 1pt} {\text{s}},0.1732{\kern 1pt} {\text{s}}]$$\end{document} , similar to other human driver modeling framework^29^. By these constraints, one single ballistic steering correction movement duration is then within 0.28 s to 0.69 s.

Human cognitive response time

In between when humans receive a stimulus to the moment of exerting a response movement, are the neurocognitive processes and mechanisms that form the pathways in the central nervous system. It may generally be understood as information propagating through (a) the perception pathway to make sense of the stimulus, then (b) the control pathway to form, plan, and predict movements, and finally, (c) the muscle pathway to execute the movements. Assuming the signal propagates through these pathways, then the total response time is defined by the sum of the cognitive processing time^29^ as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tau _{\text{ d }} \triangleq \tau _{\text{ p }} + \tau _{\text{ c }} + \tau _{\text{ m }} = t_\text {I,on} - t_\text {s,on}, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ p }$$\end{document} is the cognitive perception processing time delay, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ c }$$\end{document} the control decision time delay, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ m }$$\end{document} the muscular activation time delay for muscle recruitment (movement onset. This should match the time difference from stimuli onset \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_\text {s,on}$$\end{document} to movement onset \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_\text {I,on}$$\end{document} , see Fig. 3 for a timeline illustration for a single correction. Control decision time delay \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ c }$$\end{document} is considered general and may encompass the processes using internalized modeling such as higher cognitive functions, e.g. predictive behavior or movement planning, and may span beyond seconds before a response is initiated. Methodically characterizing this control factor is challenging due to its general nature and is heavily dependent on the internalized model and the higher-level intentions of the agent. For the purpose of this work, only immediate stimulus-triggered responses will be considered, which effectively renders \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _{{{\text{ c }}}} \approx 0{\kern 1pt} {\text{s}}$$\end{document} . In neuroscience, previous experimental results based on both humans and primates have suggested numerical estimations of delay timings to be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _{{{\text{ p }}}} \in [0.074{\kern 1pt} {\text{s}},0.142{\kern 1pt} {\text{s}}]$$\end{document} for visual perception^29,48,49^ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _{{{\text{ m }}}} \in [0.050{\kern 1pt} {\text{s}},\,0.122{\kern 1pt} {\text{s}}]$$\end{document} for motor cognition^29,50,51^.

To experimentally determine the response time (time delay) of a system, one can analyze the cross-correlation which is a common technique in signal processing in the engineering domain. To identify the system response time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ d }$$\end{document} , one finds the time delay \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau$$\end{document} that maximizes cross-correlation between input and output signals of the system as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tau _\text{ d } = \mathop {\mathrm {arg\,max}}\limits _{\tau } (C\star S)(\tau ) \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(X\star Y)(\tau )$$\end{document} is the cross-correlation (similar to convolution but without time reversal) between the control output signal C and stimuli input signal S for a time delay \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau$$\end{document} . Here a zero-normalized cross-correlation will be used

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (C\star S)(\tau ) = \rho _\text {p}(\tau ) = \frac{\text{ E }[(C(t)-\text{ E }[C(t)])(S(t-\tau )-\text{ E }[S(t-\tau )])]}{\sigma _{C(t)} \sigma _{S(t-\tau )}} \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text{ E }[X]$$\end{document} is the expected value of X and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _X$$\end{document} is the standard deviation of X. This brings the two compared signals to zero-mean and normalizes the correlation to the interval \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _\text{ p } \in [-1,1]$$\end{document} . This is also known as the Pearson correlation coefficient in statistics, which naively measures the linear relationship of the two signals.

Combining this and the intermittency property (see Eq. 8), it is possible to formulate a set of expectations regarding the latency results depending on the human cognitive processing for steering control can be characterized as online-based versus internal model-based. For the former case of online control, a strong peak should be identified in the cross-correlation, as the actions are mainly driven by readily available stimuli. This peak should be located at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _d \approx 0.15$$\end{document} as suggested by previous results in neurocognitive science. For the latter case of model-based control, it is suggested that the control process integrates previous experience and information to form the actions, i.e. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _c > 0$$\end{document} of unknown distribution, adding a heavier tail to the cross-correlation after \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _d \approx 0.15$$\end{document} .

Methods

Ethics

Informed consents were obtained from the research participants prior to the experiment to collect, process, and publish the research findings. The author depicted in Fig. 4 consented to the publication and use of the image. This research project has been reviewed and approved by the Swedish Ethical Review Authority (reference number: 2023-03453-01) in accordance with the declaration of Helsinki and the Swedish Ethical Review Act.

Research participants

A total of fourteen naive healthy adult research participants with normal or corrected-to-normal vision were recruited for the research experiment (six females, eight males, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _\text {age} \approx 25.36\,\hbox {yrs}$$\end{document} , \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _\text {age} \approx 5.29\,\hbox {yrs}$$\end{document} , ranging 18–38 yrs). Three research participants were part of the pilot phase of the study and were excluded from this work.

No formal requirements or qualifications of a driving license, past driving experience, or previous video gaming experience were imposed during recruitment. From this, a total of 110 experimental trials were procured. The research participants were also asked to provide additional information about their driving, video gaming habits, and basic demographic data through a digital survey in connection to the experiment. All participants reported to be in good health at the time of the experiment. There were some reported cases where research participants became slightly motion sick due to exposure to VR during and after the experiment. The research participants were informed that they may abort their participation at any point during the experiment without providing any explanation. All research participants were compensated with a gift card worth 99 SEK (approximately equivalent to 10 EUR) for their participation regardless of the experimental outcome.

Driving simulator

The source code repositories for the software used in the experiment to support the simulation, collect and process data, and visualize the results are made open-source and accessible^52^ through various DevOps platforms. Due to the nature of the microservice software design paradigm, the software development and code-base are simplified at the cost of having higher complexity in software deployment and runtime. However, these deployment and runtime complexities are partly alleviated through exploiting containerization deployment strategy^53^. The main desktop computer handling the computations for the experiment is equipped with an intel central processing unit (11th Gen Intel(R) Core(TM) i7-11700K@3.60 GHz) and an Nvidia graphics processing unit (RTX A2000 6 GB memory, Nvidia driver version: 515) and operated by a Linux Ubuntu 22.04 LTS operating system ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\texttt {Linux kernel 5.15.0-43-generic SMP PREEMPT\_DYNAMIC}$$\end{document} ).

Equipment and vehicle setup

An off-the-shelf head-mounted virtual reality headset system, HTC Vive Pro 1, was used to display the rendered simulation while capturing the head kinematics of the research participant in terms of translation and rotation. The head-mounted display (HMD) is capable of rendering an image of size 1440 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times$$\end{document} 1600 pixel resolution per eye at 90 Hz with a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$98^{\circ }$$\end{document} field of view. The virtual floor height was calibrated to match the real height, approximately 1.1 m from the floor to HMD in the seated position. The HMD was further complemented with a third-party eye-tracker solution system add-on, HTC Vive Binocular Add-on, from Pupil Labs (Pupil Labs GmbH, Berlin)^54^. This device captures infra-red monochrome video feed 400 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times$$\end{document} 400 pixel resolution per eye at 120 Hz of the eyes and pupils of the research participants. These eye data signals are then mapped as gaze points onto the head-fixed rendered image data.

There is an inherent display latency, a consistent time delay from rendering command to displaying the actual image on the device, of 0.018 s in the HTC Vive Pro 1 using low-level implementation in Python and OpenVR by other researchers^55^. However, it is unclear what operating system, drivers, and graphic rendering API were used for their experiment. Furthermore, the development of VR graphics rendering has made significant improvements to reduce the display latency using e.g. asynchronous reprojection. The reported display latency should be regarded as the worst-case display latency time due to the time of the publication. There is a similar challenge from the actuator sensor, the sensor signal latency may be found. Due to the scope and complexity of this work, these latencies will be considered negligible.

A vehicle hardware interface system is integrated to control the simulated ground vehicle. This system captures steering wheel signals for lateral control, and a pedal system set is used for longitudinal control. A SensoWheel steering wheel system is obtained from Sensodrive (Sensodrive GmbH, Germany) with additional mounted sensor add-ons. It can generate a fully programmable, powerful direct drive force-feedback torque towards the driver while simultaneously measuring the applied differential torque onto the steering wheel from the driver. A Logitech G27 pedal system set was repurposed for this experiment, controlling the vehicle propulsion. The vehicle hardware interface system was then mounted on top of a modified off-the-shelf racing sim cockpit (Playseat Evolution), see Fig. 4 for a view of our experimental setup with one of the authors demonstrating as a participant.

The vehicle dynamics model used in the experiment is a two-track model based on a single-seated racing vehicle as a reference with an electric propulsion system with torque vectoring applied on the rear axle. The vehicle model is paired with the magic formula tire model implementation^56^ with empirical parameters for racing vehicle tires. The complete vehicle model was designed and validated under normal driving conditions. Here the normal driving condition is characterized as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|v_y|v_x^{-1} \ll 1$$\end{document} for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v_x>0$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v_y$$\end{document} is the lateral velocity and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v_x$$\end{document} is the longitudinal velocity of the vehicle. This condition allows for smaller lateral slips to be tolerated. Transient load states and extreme lateral velocity are not handled well by the model and are beyond the scope of this research project. A complete loss of control of the vehicle invalidates the trial and that trial is omitted from further analysis. The research participants are only informed that they are operating a ground vehicle but not the particular implementation or the used vehicle model prior to the experiment.

Calibration for gaze and video analysis in virtual reality and head-mounted displays

Several calibration routines are carried out to ensure the data is harmonized and correct. One such calibration is the lens optical correction applied in the HMD. When the rendered images are shown on the display, it is intentionally pre-distorted, e.g. chromatic aberration, in the software to compensate for the optical lenses inside the HMD. Thus it is required to account for the pre-distortion of the image coordinate system when performing gaze mapping correctly in the eye tracking software. The intrinsic calibration values used in the rendering software are stored inside the firmware loaded in the HMD systems. They are individually tuned for each HMD unit during production and may be accessed through SteamVR interface console. The retrieval of the correct rectilinear projection of the rendered image in the HMD is possible by applying an inverse radial distortion camera model. The numerical intrinsic camera calibration values for our particular HMD used in the experiment can be found in Supplementary Table S1.

The eye-tracking system is sensitive to slippage of the HMD, where a minor slippage may invalidate gaze data and our analysis. The 3D-model pupil detection method was chosen by default as the model is more resilient to smaller slips^57^ which limits the effects of HMD slippage to an extent. The less sophisticated 2D-model method was used as a fallback when the performance of the default 3D-model pupil detection method deteriorated. To ensure gaze data integrity, a calibration routine was performed at the beginning of the experiment for each research participant with validation routines performed at the beginning and end of the experiment. The data performance metrics for the experimental gaze data ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=8$$\end{document} ) were on average per person, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {accuracy} \approx 0.0404\,\hbox {rad}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {precision} \approx 0.0037\,\hbox {rad}$$\end{document} using the validation samples. Gaze data samples with confidence values below 60 %, associated with blinking, or classified as outliers were discarded. Here, outliers are defined as samples falling in the extreme 20 % screen border perimeter i.e. screen normalized gaze positions x, y fulfilling the conditions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x \in ]0.2,0.8[$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y \in ]0.2,0.8[$$\end{document} are considered inliers.

Software architecture and systems integration for data harmonization

Various systems are used in this experiment, making the general data harmonization a complex task. The majority of data was interfaced, captured, and collected through our developed software framework OpenDLV using a microservice design pattern powered by the middleware software libcluon^53,58^. This allows for microsecond precision time stamping of the data signals when the signal enters the computer node with proper time synchronization across computer device nodes where such a feature is supported and requested. The data in the software framework is then serialized during the experimental trials and saved to multiple binary files for later processing.

The videos of the eyes of research participants and their gaze signals were collected and processed through the open-source software framework of eye-tracking system Pupil^54^. Our forked project has additional implementations to accommodate and process HMD-rendered image data instead of the intended head-mounted camera image feed and to account for the pre-distortion of the rendered image due to the optical lens inside of the HMD. The rendered image data in VR was captured as a X11 window texture, then stored in an interprocess communication (IPC) shared memory allowing it to be accessed by the Pupil software for further processing.

The rendering software was developed using the open-source OpenVR SDK through the Vulkan API, an open-source low-level graphics rendering API allowing complete control of the rendering pipeline. OpenVR aims to homogenize various VR-HMD hardware systems to one common software interface API to their SteamVR runtime allowing VR rendering to be displayed independently of the VR-HMD. By exploiting the Vulkan API, the rendering commands of generating the frame can be captured in gfxreconstruct and later be used to reconstruct the frame. This is accomplished by reissuing the captured rendering commands back to the rendering pipelines, which is advantageous for later analysis stages. This method only requires a fraction of file size at a minimal resource cost during serialization compared to conventional lossless compressed video data with proper encoding. Two rendered images are produced during the experimental trials for each discrete sampled time point, one for the left eye and one for the right eye. However, for simplifying the analysis only the left-eye-rendered scene images are considered in this work.

The experimental task in the virtual reality and data collection

The visual environment in VR is simplistic and kept to a minimum amount of objects: daylight outdoor scenery with blue sky and clouds, textured ground with an S-shaped two-lane road track, and a vehicle-frame-fixed gray steering wheel. The motivation for the simplistic visual environment is to make the optic flow estimation more accurate while keeping the need for post-processing at a minimum, and keeping the research participant focused on the given experimental task. The schematic details of the experiment drive track are shown in Fig. 5 and consist of an acceleration section, a left curve bend, a smaller straight section, a right curve bend, and a deceleration section. The curve bends are designed with a 60 m diameter. An example view during the left bend from the left eye of the research participant is shown in Fig. 2a.

By the design of the vehicle model, a front-axle steered vehicle traveling forward transfers a so-called aligning torque from the ground to the steering system, creating a stable fix point at neutral steering. The steering wheel motor used in the experiment features a direct drive force-feedback system and implements a simplistic aligning torque dynamic. Human drivers often use this aligning torque to their advantage by relaxing the control of the steering to straighten out the steering. This results in unnatural steering movements at the ends of curve bends. For this reason, these regions have been omitted from the analysis resulting in the middle sections of the curve bend (see the regions of interest in Fig. 5). These are sections where there is a considerable demand for steering maintenance and simultaneously natural steering corrections are performed.

Before the experimental task, calibration routines are carried out to ensure the integrity and accuracy of the data signals. Research participants are also asked to familiarize themselves with the VR environment and the user interfaces, allowing them to acclimate to the new visual stimuli from the HMD. This also serves as a good opportunity for them to explore all viewing angles from the driving seat as they are free to move their head during the experiment and adjust the fit of the HMD and the vehicle user interface if necessary. For the experimental task, the participants are asked to drive through the track on the right lane from start to end without a complete standstill while staying within their right lane at all times. They are also instructed to carry out the task at a comfortable and safe velocity of their preferred choice. The experiment is carried out without any external interruption or sudden intervention events. For the first five laps, the steering wheel is visible for the participants and then visibly removed for the remaining five laps, totaling ten trials for each participant. This condition was imposed to investigate how having vehicle-fixed objects in the visual vicinity may aid human locomotor performance.

Data was collected on the research participants performing the experimental task. The complete data set is divided into two: an open anonymized post-processed data set^59^ and a closed pseudonymized data set^60^ containing the pre-processed raw data with restricted access. These data sets may be accessed through the Swedish National Data service.

Dense retinal optic flow estimation

The perceived visual flow in the eyes is reconstructed as numerical dense retinal optic flow emerging from relative motion to the observer. This is done by fusing optic flow estimation fixed to the head of the observers and using measured gaze to account for the rapid eye movements to reconstruct retinal optic flow. For the scope of this work, some assumptions and simplifications were made to estimate the retinal optic flow. Gazing kinematic states may be classified into four distinct types: fixation, smooth pursuit, saccade, and post-saccadic oscillation. Here, all valid active fixations will be assumed and considered as smooth pursuits, where “exact” visual gaze tracking is achieved during the fixation. Valid active fixation refers to disregard gazing data sampled during and in connection to blinking and naive outlier rejection. Exact visual gaze tracking here means that the elicited gaze point precisely follows the visual feature in the scene. This implies a pursuit gain of 1.0 relative to the visual motion which results in a null flow point at the point of gaze fixation^6–8,11,16,44^. Further, it is assumed that this occurs without major slips or catch-up saccades. For further details of this idealization of smooth pursuits, see the Supplementary Materials. This assumption allows a straightforward, simplified numerical computation of retinal optic flow using the plain naive optic flow and active gaze point data as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \varvec{v}_\text {r}(\varvec{p}, \varvec{p}_\text{ g}) = \varvec{v}( \varvec{p}) - \varvec{v}( \varvec{p}_\text{ g}) \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{v}_\text {r}$$\end{document} is the retinal optic flow field, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{v}$$\end{document} the plain naive optic flow, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{p}$$\end{document} some image pixel point, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{p}_\text{ g }$$\end{document} the active gaze point in the image. One direct effect is that the optic flow vector sampled at the gaze point \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{v}_\text {r}(\varvec{p}_\text{ g }, \varvec{p}_\text{ g}) = \varvec{0}$$\end{document} is always the zero vector emulating the null visual flow at the fovea or exact retinal image stabilization. Then relating the angular direction of the vectors in the flow field is simply

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \theta (\varvec{p}) = \arctan \frac{ v_z(\varvec{p})}{ v_y(\varvec{p})} \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{v} = (v_y, v_z)$$\end{document} the vector components, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{p}$$\end{document} is a point sampled in the field. It should be noted that this uses the image space as the reference system, which needs to be accounted for in later analysis due to constant head tilts due to misalignment.

The dense optic flow in computer vision refers to a motion vector field sampled at every pixel in the data image frame, as opposed to sparse optic flow where only a few select points in the image describe flow vectors, refer to Supplementary Fig. S1 for visualization of synthetic examples of sparse and dense optic flow. There are a plethora of optic flow or scene flow (visual motion flow in three dimensions) estimation methods with varying accuracy and compute performance, especially since the recent introduction of black-box deep learning approaches^61–63^. Considering the data throughput, the accuracy-to-compute efficiency trade-off, and hardware-accelerated computation capabilities, Nvidia optic flow SDK 2.0 is used in our work to estimate the optic flow. This method is accessible through the open-source computer vision library, OpenCV. However, its implementation is not fully disclosed and may vary depending on the generation of the Nvidia graphics card due to the optic flow estimation implementation being tied to the hardware architecture. Our source code for computing the dense retinal optic flow is made openly available^52^.

Each visual view frame of the scene from the HMD of the research participant was reconstructed and processed losslessly from the left eye. The post-rendering of the scene was carried out without the steering wheel to simplify the image post-processing in the analysis, such as the lane detection segmentation and optic flow estimation. The dense retinal optic flow could then be reconstructed by combining the dense optic flow and the gaze data from the eye-tracker system, see Fig. 2c for retinal optic flow field visualization.

Assuming there is no slippage of the HMD during the experiment, there is a significant constant angular tilt bias in the visually rendered image. There are many contributing factors to this, e.g. the mounted HMD is slightly misaligned or research participants naturally have their head slightly tilted during the experiment. This angular head tilt directly affects the retinal optic flow angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bar{\theta }}_\text {rof}$$\end{document} which is derived from the image space as the reference coordinate system. To mitigate this and make the results comparable across participants, the median of the retinal optic flow angle (constant bias) using the samples during the straight sections of the drive track is subtracted from the retinal optic flow angle samples. This re-centers the data around zero across research participants, thus making the retinal optic flow angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bar{\theta }}_\text {rof} \approx 0.0$$\end{document} during the straight drive sections. This angular tilt bias was small for the majority of the participants. However, some participants had a significant tilt of their view in the HMD skewing the retinal optic flow angle measures.

Using particle swarm optimization to identify ballistic corrections through decomposition of the sum of Gaussian functions

Naturalistic limb movements, known as reachings, are composed of complex movement patterns through overlapping simpler ballistic corrections. These reaching corrections are often bell-shaped in their velocity profiles^34^ and may be approximated by Gaussian functions. Thus the general limb movements may be decomposed into their constituent corrections with varying parameters of strength \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a_\text{ I }$$\end{document} , mode \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _\text{ I }$$\end{document} , and rate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _\text{ I }$$\end{document} parameters as described in Eq. (5) and the resulting limb movement in Eq. (6).

This challenge of decomposing a sum of Gaussian functions is not specific to the domain of biomechanics. It is also found in the scientific and engineering fields of astronomy, spectroscopy, signal processing, data compression, and image processing. Thus, there are existing methods to decompose the composed signal with varying results depending on the emphasized metrics, for example, scale-space imaging using Levenberg-Marquardt algorithm^64^, deconvolution techniques^65^, and iterative method for decomposing a sum of Gaussian functions^66^. Since researchers from various fields have attempted to design their algorithms, they utilized varying assumptions and constraints that might only apply to their particular field. Such examples could be a priori knowing, how many constituent Gaussian functions the signal should be decomposed into, or whether the Gaussian components are strictly positive, etc.

This work developed and implemented a meta-heuristic optimization method algorithm, particle swarm optimization (PSO), to decompose the observed signal to its approximate individual constituent Gaussian contributions. Meta-heuristic or evolutionary algorithms are methods inspired by biological processes to solve various problems by iteratively optimizing to converge to a candidate solution to an objective or fitness function. Major advantages of using such meta-heuristic algorithms as opposed to classical optimization methods are that they generally scale better with regards to dimensions of the search space, do not need to explicitly compute the derivatives of any order of the objective function, and are comparably robust to measurement noise^67^. However, meta-heuristic optimization methods including the PSO algorithm, perform exhaustive searches in the well-defined search space and do not guarantee the candidate solution to be the global optima. Optional a priori constraints may be imposed upon the search space to induce quicker convergence and to reject false solutions.

To experimentally determine the ballistic corrections, assume the observed signal with a train of correction can be described as a sum of Gaussian functions, as described in Eq. 6, then the objective or fitness function to solve for a given time interval becomes

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Omega _{{\dot{\delta }}}^* = \mathop {\mathrm {arg\,min}}\limits _{\Omega _{{\dot{\delta }}}} \frac{N_{\dot{\delta }}^\alpha }{t_1-t_0} \int _{t_0}^{t_1} |z(t) - \sum _{(a_i,\mu _i,\sigma _i) \in \Omega _{{\dot{\delta }}}} {\dot{\delta }}_\text{ b } (t,a_i,\mu _i,\sigma _i)| \text{ d } t, \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega _{{\dot{\delta }}}$$\end{document} is a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3\times N_\delta$$\end{document} matrix where each column in the matrix contains the parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(a_\text{ I }, \mu _\text{ I }, \sigma _\text{ I})$$\end{document} for a single ballistic correction, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} is a parameter to balance data fitting (to avoid overfit), and z(t) is the experimentally observed signal. PSO numerically solves Eq. 16 yielding a candidate solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega _{{\dot{\delta }}}^*$$\end{document} which is used for the analysis. Further PSO implementation details are provided in the Supplementary Material.

Results

A total of fourteen research participants were recruited for the experiment. Three of the fourteen research participants were part of the pilot experiment and thus excluded from this work. 110 trials were procured from the eleven research participants, of which 104 were considered successful. An example of a failed trial would be a loss of vehicular control such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|v_y| \gg |v_x|$$\end{document} . The trial outcomes from the eleven participants are shown in Table 1. Only eight participants completed ten successful trials out of the possible ten. These particular eight research participants are considered for the main analysis in this work.

Some participants commented after the experiment that they experienced the trials more difficult to complete when the steering wheel was visibly removed. This is reflected in our results where unsuccessful trials occurred more often without the steering wheel. The common point of failure was ultimately losing control of the vehicle and swerving out of the designated lane. This is mainly due to strong rapid steering corrections applied at high speed resulting in the loss of tire traction.

Reconstructing and quantifying retinal optic flow stimulus in curve bends

The retinal optic flow angle was computed over the lane segment region for every rendered frame, as detailed in Eq. (1) and demonstrated in Fig. 2. Figure 6 shows the time series of experimental data such as the steering wheel angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta$$\end{document} , retinal optic flow angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\bar{\theta }}_\text {rof}$$\end{document} , heading deviation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _\text{ h }$$\end{document} and lateral displacement \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_\text {l}$$\end{document} for one complete trial for four different research participants. An interesting observation is that the vehicular heading deviation oscillates around zero which is an essential outcome of intermittent control for successfully maintaining proper steering.

The retinal optic flow angle may look noisy at first glance. However, this is an artifact from active gaze fixation intermittency where the participant actively shifts their fixation around their driving lane within the distance headway \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_\text{ h }\in [4\,\hbox {m},25\,\hbox {m}]$$\end{document} (euclidean distance to gaze point on ground from the eye participant) or the time headway \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{{{\text{ h }}}} \in [0.4{\kern 1pt} {\text{s}},2.5{\kern 1pt} {\text{s}}]$$\end{document} (distance headway normalized by the instantaneous vehicular velocity). Compared to the literature, the time headway results presented here slightly deviate partly because of a lower head position to the ground, since the participants are situated lower than in a standard passenger vehicle. Furthermore, the time headway quantity depends on the vehicular velocity, as higher velocities result in lower time headway values. Interestingly, although the research participants were free to control their longitudinal velocity on the condition they could safely execute the task, the majority still drove at a high velocity. The average velocity of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v_\text{ l } \approx {13}\,{m\,s}^{-1}$$\end{document} in the curve bends, which corresponds to a lateral acceleration of approximately \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$5.63\,\hbox {m\,s}^{-2}$$\end{document} (equivalent 0.57 g-force). This can partly be explained by the fact that the research participant did not have access to a speedometer during the trials. This may have resulted in them relying solely on their visual perception to subjectively misjudge their traveling speed.

In the curve bends, the consistent guiding fixation behavior was emitted by the large majority of research participants. In detail, the exact placement of the guiding fixation was often placed in the neighborhood of their designated lane, more often in the inner parts of the curve around the apex line, see Fig. 7 for the gaze distribution during the curve bends. This may reinforce the idea that proper gaze control is necessary to fully exploit the retinal optic flow strategy for properly maintaining steering.

Identifying ballistic steering corrections

From the 80 trials from the eight research participants, a total of 2084 individual ballistic intermittent steering corrections were identified in the curve bends, see Table 2. Figure 8 demonstrates the identified ballistic corrections for four research participants for selected time intervals. The correction strength \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a_\text{ I }$$\end{document} , mode \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _\text{ I }$$\end{document} , and rate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _\text{ I }$$\end{document} parameters are estimated for each correction, reconstructing the resulting reaching movement as a whole using the principle of superposition as described in Eq. (6). The larger discrepancies of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\dot{\delta }}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\dot{\delta }}_\text {fit}$$\end{document} at the ends of corrections, seen in Fig. 8b at approximately 166 s and 166.8 s for an example, are most likely an effect of the aligning torque exerted as the direct drive force-feedback to the participant.Fig. 1. Retinal optic flow and retinotopic flow. The illustration demonstrates how visual motion flow is projected to the retina in the eye, creating the retinal optic flow or retinotopic flow sensation the observer experiences. The retinal optic flow field accounts for the rotational eye movements exhibited during gaze fixation and smooth pursuit, achieving stabilization of the retinal image. The main difference between retinal optic flow and retinotopic flow is the stereographic projection, i.e. projection to the sphere skewing coordinate system.Fig. 2. Visual input, image processing, and data visualization. Panel (a) Demonstrates the point of view of a research participant in virtual reality. The virtual world is made visually simple. It consists of a textured sky, textured ground with painted lane markings, and a steering wheel. This particular time instance is captured during the left curve bend negotiation. Panel (b) shows the post-processed data rendered on top of the raw image view without the steering wheel. In the image, the gaze point of the participant is illustrated with a black crosshair, and the detected lane segment is colored as a green area. Panel (c) shows retinal optic flow stimulus is reconstructed by combining motion analysis of temporal consecutive images and gaze data. The colors represent the direction of the optic flow vectors in the field, see Supplementary Fig. S1 for further detailed explanation and illustration. The green color is equivalent to the retinal optic flow pointing in the downward direction. For visual reference, the lane markings are painted white on top of the flow for this illustration. Videos of the author driving are linked here: https://youtu.be/rW_X9OZ3mRM and https://youtu.be/d0xtk2d4wFk (with GUI).Fig. 3. Intermittent control timeline for a single correction. The figure illustrates the timeline of an intermittent control where a ballistic correction is triggered by a stimulus at the stimulus onset time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_\text {s,on}$$\end{document} . The perceived need for an action is highlighted with gradients of red. There is also a small time window to sample the perceived error, a process known as evidence accumulation. Once this perceived error becomes large enough, a ballistic correction is triggered as a response at the time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_\text {I,on}$$\end{document} . If the expressed response is correct and proper, it reduces the need for action.Fig. 4. The experimental hardware setup with one of the authors demonstrating as a participant. The image depicts a fully assembled experimental setup consisting of a virtual reality head-mounted display with an eye-tracking system (not visible), a racing-sim chair with a mounted steering wheel rack with a direct drive force-feedback feature, and a pedal set system. There are also VR base stations in the room tracking the head movements of the research participant.Fig. 5A schematic of the track design used in the experiment. The dimensions and the blueprint of an S-shaped two-lane road track are explicitly shown in Figure. The track contains straight road sections with left and right semi-circular curve bends with a diameter of 60 m. The straight road sections are created for the research participants to accelerate and decelerate to a velocity of their choice. The lanes are defined with white lines wholly drawn, with a double line in the middle separating the left and right lanes. For the analysis in this work, only the regions of interest are considered. It should be noted that the right lane in the regions of interest of left and right curve bends differ in length and radius due to being situated in the outer and inner bends.Fig. 6. Experimental time series data for one complete trial for four research participants driving in an S-shaped track. The figure shows time series data excerpts of four different research participants driving with a visible steering wheel in panels (a) and (b), and without a visible steering wheel in panels (c) and (d). The upper plots in each panel show the experimental time series of the steering wheel angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta$$\end{document} and retinal optic flow angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{\theta }_\text {rof}$$\end{document} . The driving metrics vehicular lateral position \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_\text{ l }$$\end{document} and vehicular heading \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _\text{ h }$$\end{document} relative to the lane are shown in the lower plot for the same time interval. Vehicular heading deviation is related to the time derivative lateral displacement explaining its close relation. The trial consists of a left-curve bend followed by a right-curve bend, which can be identified in the steering wheel angle. Compared to the naive optic flow case, the retinal optic flow appears to be noisy but the transients are the results of the gaze fixation intermittency.Fig. 7. Human gaze distribution around the curve bends. The figure shows the gaze distribution as a heat map of the eight research participants for the left curve (upper rows) and right curve (lower rows). The research participants tend to fixate their gazes on the inner parts of the curve bends, close to the apex.Fig. 8. Experimentally identified intermittent ballistic corrections in the steering wheel. The steering wheel angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta$$\end{document} and derived angle rate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\dot{\delta }}$$\end{document} during the curve bends from the experiment are shown here for various research participants. Using particle swarm optimization, our analysis method identifies the ballistic reaching corrections \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\dot{\delta }}_\text {b}$$\end{document} in the steering wheel angle rate by approximating the correction strength \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a_\text{ I }$$\end{document} , mode \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _\text{ I }$$\end{document} , and speed rate parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _\text{ I }$$\end{document} . The stems represent the strength and mode of each correction. The resulting fitted steering wheel angle rate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\dot{\delta }}_\text {fit}$$\end{document} , the sum of each theoretical ballistic correction \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\dot{\delta }}_\text {b}$$\end{document} , is comparable to the observed experimental data. Each correction onset timing is indicated for each ballistic correction, as defined in Eq. (8).Fig. 9. Absolute steering ballistic correction angle distributions. The figure shows the absolute steering ballistic correction angle distributions for the eight research participants who succeeded in all ten trials out of ten possible. The distribution for trials with visible steering wheel is shown in panel (a), and the distribution for trials without visible steering wheel in panel (b). Panel (c) shows the distribution using data combined from (a) and (b). All distributions generally follow a theoretical log-normal distribution with similar parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\log \mathcal {N}(\mu = 0.08, \sigma ^2 = 0.66)$$\end{document} in this work. Visibly removing the steering wheel does not seem to impact the ballistic steering correction angles significantly.Fig. 10. Response time analysis using zero-normalized cross-correlation. The figure shows the cross-correlation between stimuli and steering wheel correction angle over response time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ d }$$\end{document} for trials with a visible steering wheel in panel (a) and without a visible steering wheel in panel (b). The cross-correlation is computed using the Pearson correlation coefficient. The response time is investigated for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _{{{\text{ d }}}} \in [0{\kern 1pt} {\text{s}},2{\kern 1pt} {\text{s}}]$$\end{document} , measured from the ballistic steering correction onset to the stimuli onset as described in Eq. (11). The strongest cross-correlations with respect to response times for the stimuli are indicated by black vertical bars. According to the literature, the red vertical bar indicates the approximate minimum human response time directly triggered by a stimulus. The color-shaded time intervals indicate the neighborhood maximum of the cross-correlation for the two stimuli. The strongest cross-correlation between the retinal optic flow angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{\theta }_\text {rof}$$\end{document} and the steering wheel correction angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta \delta _\text{ b }$$\end{document} is found approximately at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ d } \in \{0.14\,s,0.32\,s\}$$\end{document} . Equivalently for the vehicular heading \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _\text{ h }$$\end{document} , strongest cross-correlation was found \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ d } \in \{0.44\,s,0.60\,s\}$$\end{document} . Removing the visible steering wheel significantly increases the response time with approximately 0.17 s and widens the time window of the retinal optic flow angle making the steering corrections responses applied more inconsistently. There is a significant trailing tail to retinal optic flow angle when the steering wheel is visibly removed, suggesting a predictive behavior. The data views with corresponding scatter plots where the strongest cross-correlations are shown in Fig. 11.Fig. 11. Strongest correlation at selected response time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ d }$$\end{document} of stimulus predictors to ballistic steering correction angle. The figure shows the relationship of the observed signals where the strongest correlation is found. Here the stimulus predictors are retinal optic flow angle and vehicle heading angle deviation, and the resulting output control signal is the ballistic steering correction angle. The panels (a) and (b) show the observation made for the successful trials with a visible steering wheel, while the panels (c) and (d) show the equivalent but without a visible steering wheel. The frequency histograms for the observed data are shown next to the scatter graphs for the respective axis. A naive Pseudo-Huber loss linear regressor is applied to the observed data approximating the linear relationship \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{\varvec{\beta }}}$$\end{document} . The retinal optic flow angle presented here \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta {\bar{\theta }}_\text {rof}$$\end{document} has been centered around the respective mean angle values for left and right curve bends sections to only look at the incremental values.

Table 1. Experiment trial outcomes.Visible steering wheelNo visible steering wheelTotalSuccess5351104Failure246Total5555110Failure occurrence3.6%7.3%5.5% The table presents the trial outcomes for eleven research participants participating in five trials with a visible steering wheel and five trials without a visible steering wheel. Here, a success trial is defined as the participants reaching the end of the track without ever fully exiting the designated road lane. For example, excessive vehicle skidding resulting in a loss of vehicle control and sliding out of the lane would constitute a trial failure. The trial failures occurred more often without a visible steering wheel. Table 2. Identified steering corrections in the curve bends.Visible steering wheelNo visible steering wheelTrial0123456789TotalRp 034211819152015231315193Rp 133272832313030322934306Rp 226363222233030302624279Rp 327212425312421262624249Rp 426231618201622231826208Rp 526322532212622232526258Rp 638373427262426242027283Rp 728373828223532322630308#Corrections2382342152031892051982131832062084#Trials888888888880Avg. Corr./Trial29.7529.2526.8825.3823.6325.6324.7526.6322.8825.7526.05Avg.26.9825.1326.05The table presents the total number of identified corrections and the average number of corrections per trial for eight research participants with ten successful trials out of possible ten. Behavioral acclimatization or a learning effect of using repeated trials can be seen from trials 0 to 4. There is a significant trend in steering behavior where the number of corrections decreases for each lap until the fifth lap. Visibly removing the steering wheel does not significantly impact the number of corrections for each trial.

Visually removing the steering wheel does not significantly affect the characteristics of steering corrections

The resulting trial outcomes are shown in Table 1 for the eleven research participants who participated in the experiment. The unsuccessful trials occurred more often when the steering wheel was visibly removed.

The steering behavior quantified by the number of corrections per trial lap is shown in Table 2. A clear acclimatization or learning effect can be observed in the exhibited number of steering corrections. This could be due to repeated use of identical trials or the research participant learning how to handle the vehicle better. The number of exhibited steering corrections decreases from Lap 0 to 4 (from 238 to 189). This trend cannot be observed for the remaining laps 5 to 9 with no visible steering wheel. Furthermore, visibly removing the steering wheel did not significantly impact the number of steering corrections.

Another characteristic is to look at the absolute correction angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\Delta \delta _\text{ b }|$$\end{document} as defined in Eq. (10). The distributions of the absolute correction angle for the eight participants are shown in Fig. 9 aggregated for trials with, without a visible steering wheel, and all combined. The observed distributions follow a log-normal distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\log \mathcal {N}(\mu = 0.08, \sigma ^2 = 0.66)$$\end{document} regardless of the visibility of the steering wheel.

Response time of intermittent steering maintenance

So far, results of quantifying the retinal optic flow and identifying ballistic steering wheel corrections have been presented. However, the question remains if human locomotor control in curvilinear motion exploits retinal optic flow for properly maintaining steering can be observed in high-quality experimental data. Assuming that some input predictors drive output reaction responses, then every correction time onset \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_\text {I,on}$$\end{document} should occur after predictor stimuli onset \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_\text {s,on}$$\end{document} by a delayed response time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text {d}$$\end{document} due to cognitive processing.

Figure 10 shows the cross-correlations using the Pearson correlation coefficient \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _\text{ p }$$\end{document} with respect to some response time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _d$$\end{document} for the predictor variables retinal optic flow angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{\theta }_\text {rof}$$\end{document} and vehicular heading deviation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _\text{ h }$$\end{document} to the response variable ballistic steering wheel correction angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta \delta _b$$\end{document} . The human response time should be found at the strongest cross-correlation, see Eq. (11). The strongest cross-correlation between retinal optic flow angle and individual steering wheel correction angle is found at approximately \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _{{\text{d}}} \approx 0.14{\kern 1pt} {\text{s}}$$\end{document} with visible steering wheel and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text {d}\approx 0.32\,{\text{s}}$$\end{document} without visible steering wheel respectively, see Fig. 10a and b. Equivalently for the vehicular heading deviation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _\text {h}$$\end{document} , the strongest cross-correlation is found at approximately \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text {d}\approx 0.44\,{\text{s}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text {d}\approx 0.60\,{\text{s}}$$\end{document} respectively. From this, an additional latency of approximately 0.17 s could be observed when the steering wheel was visibly absent.

Figure 11 shows data views of the cross-correlation at the specific values of time lag \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ d }$$\end{document} where cross-correlation was found the strongest for the predictor variables. It illustrates the observed relationship between the predictors and steering wheel correction angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta \delta _\text{ b }$$\end{document} and their respective distributions. The retinal optic flow angle has been recentered around its respective mean values for left and right curve bends, thus only studying incremental angle values here. A naive Pseudo-Huber loss linear regressor has been applied to the observed data, determining the regression coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{\varvec{\beta }}}$$\end{document} to visualize the linear relationship better if such exists.

Discussion

In this work, we investigated the retinal optic flow (combined with lane segmentation) and vehicular heading as a potential source of information for properly maintaining steering during curvilinear locomotion. This was done in conjunction with characterizing human lateral control as an intermittent control process where steering wheel adjustments are regarded as a ballistic correction in the form of reaching movements, with an associated strength and speed. Intermittent control makes it possible to characterize each correction with a well-defined response onset in the experimental data. A time delay analysis could be performed using cross-correlation measures. This is done by studying each ballistic control correction to retinal optic flow angle-based cues and heading deviation-based cues with respect to response time as described in Eq. (12).

The response time estimated from the retinal optic flow angle with a visible steering wheel was significantly lower than the more widely used 0.20 s. This may be explained by our definition of response time. Namely, the timestamps used for response time were measured from stimulus generation onset to correction movement onset, see Fig. 3. In this way, most of the time spent on actuating the movement is excluded from the measurement. In contrast to other experiments, it is common to include the movement travel time to the trigger of signal onset, for example, pressing a button or moving a limb above a threshold. If the movement onset in this work would instead be defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_\text {s,onset} = \mu _\text{ I } - \sigma _\text{ I }$$\end{document} (at 13.6 % of the correction movement) then the discrepancy difference would be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _\text{ I } \approx 0.071\,{\text{s}}$$\end{document} bringing our reported values more in line with earlier studies reporting 0.20 s. This would make the response time measurements less consistent (as it would also depend on the movement travel). Recent research in mental chronometry report response timings closer to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text {d} \approx 0.15\,{\text{s}}$$\end{document} where the limb movements are excluded^21,25,29,51,68,69^. This is outlined in Fig. 11 to be used as a reference. Our definition of response time limits the influence of the reaching time on the response time, making the timings more consistent regardless of the strength and rate of the correction. One may argue onset time to be defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_\text {s,onset} = \mu _\text{ I } - 3\sigma _\text{ I }$$\end{document} (at 0.1 % of the correction movement) to further limit the influence of the movement on the timings effectively. However, there is also an inherent trade-off to balance; where the lower threshold is chosen, the more sensitive estimation becomes to numerical error due to computational approximation. If this is a concern, then an alternative definition may be defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_\text {I,on}= \mu _\text{ I }$$\end{document} at the inflection point, effectively making it independent of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _\text{ I }$$\end{document} .

Spearman’s rank correlation coefficient and Kendall rank correlation coefficient as an alternative measure of cross-correlation were investigated and applied in this work. The motivation is that the relationship between the stimuli predictor and the output response may not be linear, thus potentially making the Pearson correlation coefficient an unsuitable approach. Although the strength of cross-correlation varied among the methods, the general trends and shapes remained similar regardless of the method of computing the cross-correlation. For this reason, the results presented here use the Pearson correlation coefficient.

The other main finding was that visually removing the steering wheel increased the response time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ d }$$\end{document} of applying steering correction, further adding an approximate delay of 0.17 s which can be observed by comparing results in Fig. 10ba to Fig. 10b. At the same time, it did not significantly impact the steering correction characteristics in terms of the number of corrections or the ballistic correction angle length \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\Delta \delta _\text{ b }|$$\end{document} as seen in Fig. 9. This suggests that having multiple modes to perceive different aspects of egomotion may improve locomotor performance, in this case, the timeliness (lower latency) of the applied steering correction.

There is an identifiable retinal optic flow angle peak in Fig. 10a suggested and characterized by online control. This is the definition of online-driven control where the informed control adjustments are based on readily available information. On the other hand, for model-based control, anticipatory responses are challenging to study in experiments with human participants, especially when the flow of information is continuous. Anticipatory responses are based on some previous stimuli information (possibly from other sources) combined with some internal models producing the predictive control. Implied by this, these have no typical control characteristics than a non-trivial contribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ c } > 0$$\end{document} . This could explain the heavy tail observed in the response time for the retinal topic flow angle seen in Fig. 10b and the larger time window when the steering wheel was visibly absent.

Regarding the debate on what perceptual cues may be exploited for maintaining steering in curvilinear paths, our results suggest significantly different time regimes for retinal optic flow angle and heading respectively. This is under the premise that these perceptual cues drive the perceived need to produce a locomotor correction response to achieve the task. There is also the possibility that these perceptual cues are simultaneously used for locomotor control. Implied by the results, suggests that retinal optic flow may operate on a closer time horizon compared to that of heading. Effectively, this makes corrections based on retinal optic flow more suitable for transients, while corrections based on heading deviation are for relatively long-term steering maintenance.

Our implementation of reconstructing the retinal optic flow and visually analyzing the lane segmentation shares similarities with other computational perception-driven human driver models, such as the two-point control model^70^. In short, the two-point control model samples a near-point to control the lateral positioning and a far-point for heading control. Retinal optic flow angle achieves similar results but through a different method of sampling information. The lateral displacement directly affects the lane placement in the visual periphery, which in turn dominantly affects the sampling set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega _\text {lane}$$\end{document} determining the retinal optic flow angle, see Fig. 2b. In our particular implementation, the samples are equally weighted in the computation of the retinal optic flow angle. However, these may be weighted non-uniformly to emphasize different parts of the lane segment to achieve better driving performance or emulate other known biological features.

Visibly removing the steering wheel significantly impacted locomotor performance when looking at the response timings. However, the number of applied steering corrections and the distribution of correction travel angle \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\Delta \delta _\text{ b }|$$\end{document} remained unaffected by the change. However, the response time was increased \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ d }: 0.14\,{\text{s}} \rightarrow 0.32\,{\text{s}}$$\end{document} for the retinal optic flow angle and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ d }: 0.44\,{\text{s}} \rightarrow 0.60\,{\text{s}}$$\end{document} for vehicular heading deviation (roughly an added latency of 0.17 s). This is partially supported by the comments made by the research participants after the trials, stating that they experienced increased difficulty in completing the trials when they could no longer see the steering wheel. However, it remains unclear what of the following underlying mechanisms contribute to the delayed response: (a) the direct visual feedback of the immediately applied steering correction, (b) having the steering wheel visibly rotating with the real one in aiding the perceived steering or heading direction, (c) perceiving the position of the steering wheel object aiding the perceived lateral position relative to the lane, or even (d) a combination of these. One possible hypothesis is that humans utilize the extra source of information to reaffirm internal models and by extension the ego-state estimation. Having additional modes of perceptual cues could allow for a tighter, timelier, and more accurate intermittent control correction, resulting in improved performance of steering maintenance.

Our experimental findings and interpretations are coherent with the computational framework of human driver behavior for maintaining lateral steering in curvilinear paths^29^. This is however not the case during the curve negotiations when entering and exiting the curve bends. Our cross-correlation measures for predictors to control correction rapidly diminish when including the entrance and exit regions of the curve bends, see Fig. 5. The model using retinal optic flow angle as a primary source for a universal steering control is insufficient. This might suggest different perception-control mechanisms may be at play in these contexts, e.g. predictive control or retrieval of heading, thus these driving regions should be studied separately.

Our contribution demonstrates the convergence and harmony of different theories made in isolation from various research fields. In particular, contributions from visual perception and cognitive information processing to muscle coordination and motor control in humans have been utilized, as well as contributions from engineering fields such as computer vision and signal processing. Here it is shown that using a unified model to describe human locomotor behavior is possible.

Human locomotor behavior as a sustained sensorimotor control with intermittent responses

Our work emphasizes and demonstrates that human steering behavior is an intermittent process where humans intermittently apply incremental ballistic steering corrections to maintain their path. Although the idea is not novel^27–29^, this study makes full use of intermittent control to describe the ballistic action response. This allows us to study the human locomotor behavior as a closed loop from visually perceived stimuli to a ballistic action response

Researchers have long debated whether human behavior in a control context can be described as either strong online or model-based^18^, and further what and how perceptual information is used for the control mechanism^12^. Human behavior is a complex convoluted system with many different intermittent, concurrent, and asynchronous interplaying cognitive processes. One example of this is that human drivers are not only tasked to maintain sufficient lateral control of the vehicle but also to plan the coming long-term navigational decisions, maintain proper longitudinal velocity, and be aware or alerted to the surroundings. This is further reinforced by the fact that gazing itself during driving does not always lead to lateral control^71^. We argue that the human chaotic gazing behavior is an attentive monitoring process to intermittently sample relevant information in order to judge the need to correct, a satisficing behavior.

Our results may provide additional value and insight to the debate of human sensorimotor control as a strong online-driven or model-based^18^. Previous research findings for a strong online-driven control^21,29^ such that responses approaches \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _\text{ d } \approx 0.15\,{\text{s}}$$\end{document} . Furthermore, it is implied that humans complete the muscle control process before the neurocognitive processes of attention, planning, and prediction are fully completed. One interpretation of this phenomenon is that humans are not completely aware of their actions until the action itself has been carried out^24,72,73^. Another interpretation is that the control is both reactive and corrective. An initial correction response is first issued, and if it is insufficient or does not match the intended anticipated outcome, additional corrections are applied, as proposed in the other works^29^. This offers an explanation for the observed trains of overlapping corrections in the steering wheel, see Fig. 8c for example.

In our results, when the steering wheel was visibly absent, the participants may have produced anticipatory responses to maintain their curvilinear paths. This strategy may have been a coping strategy as the participants lost a reliable source of visual information used to update and monitor their steering corrections. This may explain the observable increase in the response time in our results. It has been previously shown that humans use predictive gaze strategies in the driving context when presented with predictable patterns of waypoints to sample information for the next course of action^17^. Similarly, human perception tends to find patterns to make informed predictive guesses to carry out actions appropriately^74^. For humans, forming cognitive internal models seems to be a central component of visualizing, achieving, and carrying out the task through consequential analysis; to accomplish that, this needs to be done. This process could be an evolutionary artifact to condition human behavior to successful and desirable outcomes based on memories and past experiences.

Yet, at the same time, it would seem that cognitively humans initiate actions before the planning and prediction processes are completed, or surprisingly are fully aware of their actions themselves. It is yet unclear how exactly humans can instinctively and hastily perform the initial response without having a complete plan and fully mapped actions to execute. One hypothesis is that the response itself is not required to be critically correct, but the emphasis on its timeliness with sufficient accuracy of the rapid corrections^22,23,25^. Would the rapid response not lead to the desirable outcome or align with the internalized prediction model, through a process called neuronal evidence accumulation^29^, an additional incremental correctional responses are applied on the fly. This process may also make use of immediate available perceptual information to quickly update internal models of which to enact a newly (re)weighted response to achieve the task^3,25^.

In summary, human locomotor control should be understood as a hybrid control with intermittent correction responses making use of both immediately readily available information and internal models to account for anticipation and prediction to monitor previously applied corrections. This enables the human to internalize the most favorable outcome in addition to monitor and continuously evaluate the self-perceived ego-state. Furthermore, human control modeling should be analyzed and viewed as an intermittent control process, where ballistic corrections are continuously and asynchronously applied to steer it to a targeted goal.

Conclusions

This work investigated how retinal optic flow and direction can serve as a perception cue to maintain steering for curvilinear paths. Visually rendered images and vehicular data were collected from eight adult research participants driving in an S-shaped drive track in a texture-rich environment in virtual reality. In data analysis, the retinal optic flow was reconstructed with the help of an eye-tracking system that accounts for the gaze dynamics. Retinal optic flow and lane segmentation, from the readily available visual environment, enabled a perception quantity of angular directional for further analysis for human locomotor control in curvilinear paths. The lateral control during the curve bends exhibited by the research participants was studied as intermittent ballistic reaching movements, similar to the naturalistic reaching movements in limbs. Each ballistic steering wheel correction was identified using PSO to estimate the correction mode, the correction rate parameter, and the correction strength. The impact on driving performance when having a vehicle-fixed steering wheel visible was also investigated.

The data, from the eight research participants who successfully completed ten trials out of ten possible, reveal significant cross-correlational measures between directional ballistic steering correction angles for retinal optical flow-based and heading-based stimuli at different response time regimes. It was previously known that emerging human gaze behavior achieves retinal image stabilization. However, we argue through our findings that it has applications beyond solely image stabilization, for human locomotor control in curvilinear motion. The placement of the gaze and the fixation provides an additional mode to perceive the need for control or evidence accumulation. Furthermore, under normal driving conditions with a visible steering wheel, the time delay analysis reveals that the correlation between the retinal optic flow-based cue and the steering correction response is the strongest at approximately 0.14 s from the onset of sensation to the onset of movement. Simultaneously for the heading-based stimuli cue, the response time is found at 0.44 s (compared to the reported 0.15 s response time according to other literature).

It was also found that hiding the vehicle-fixed steering wheel significantly delayed the initiation of the ballistic steering correction. This delay was observed to be approximately 0.17 s. However, the absence of a visible steering wheel did not substantially affect the characteristics of the corrections in the steering wheel: the number of corrections or the angle length of the steering correction.

The consistency in response time suggests an online control for steering maintenance during curvilinear paths. At the same time, when visibly removing the steering wheel indicates reliance on predictive control or anticipatory responses for the retinal optic flow-based cue, arguing for model-based control may be at play. This is in line with the sustained sensorimotor control model with intermittent decisions of human locomotor behavior, which is a hybrid approach using both online control and model-based control. Thus, human locomotor control behavior should be understood as a system with various overlapping cooperative and coordinated cognitive processes that ultimately accomplish a task through a collaborative and collective effort.

Supplementary Information

Supplementary Information.