Context-Aware Recursive Bayesian Graph Traversal in BCIs

TL;DR

This paper introduces probabilistic graphical models that incorporate context and EEG evidence to improve decision accuracy in noninvasive brain-computer interfaces, especially for users with poor calibration performance.

Contribution

It proposes two novel probabilistic graphical models utilizing context and EEG data for enhanced graph-based decision-making in BCIs, demonstrating improved performance.

Findings

Probabilistic models outperform traditional methods in low calibration scenarios.

Context-aware models achieve comparable results to high calibration users.

Simulation shows significant performance boost with the proposed approach.

Abstract

Noninvasive brain computer interfaces (BCI), and more specifically Electroencephalography (EEG) based systems for intent detection need to compensate for the low signal to noise ratio of EEG signals. In many applications, the temporal dependency information from consecutive decisions and contextual data can be used to provide a prior probability for the upcoming decision. In this study we proposed two probabilistic graphical models (PGMs), using context information and previously observed EEG evidences to estimate a probability distribution over the decision space in graph based decision-making mechanism. In this approach, user moves a pointer to the desired vertex in the graph in which each vertex represents an action. To select a vertex, a Select command, or a proposed probabilistic Selection criterion (PSC) can be used to automatically detect the user intended vertex. Performance of different PGMs and Selection criteria combinations are compared over a keyboard based on a graph layout. Based on the simulation results, probabilistic Selection criterion along with the probabilistic graphical model provides the highest performance boost for individuals with pour calibration performance and achieving the same performance for individuals with high calibration performance.

Figures

Equations

$$\begin{array}{rl}& P(s_{e_t}|\{{\mathbf{X}}_{e_i}{\}}_{i=1}^{t-1},{\mathbf{X}}_{e_t},w_e,L)\propto P({\mathbf{X}}_{e_t}|s_{e_t})\sum _{y_{e_t}}P(s_{e_t}|y_{e_t})\\ & \times \sum _{A_{e_t}}\left[P(y_{e_t}|A_{e_t},w_e,{\{{\mathbf{X}}_{e_i}\}}_{i=1}^{t-1},L)P(A_{e_t}|L)\right]\end{array}$$

$$\begin{array}{rl}& P(s_{e_t},T_e|{\{X_{e_i}\}}_{i=1}^{t-1},X_{e_t},w_e,L)\\ & \propto P(s_{e_t},T_e,X_{e_t}|{\{X_{e_i}\}}_{i=1}^{t-1},w_e,L)\\ & =P(X_{e_t}|s_{e_t})\times P(s_{e_t},T_e|{\{X_{e_i}\}}_{i=1}^{t-1},w_e,L)\\ & =P(X_{e_t}|s_{e_t})\times P(s_{e_t}|T_e)\\ & \times P(T_e|{\{X_{e_i}\}}_{i=1}^{t-1},w_e,L)\end{array}$$

$$\begin{array}{rl}& P(T_e|{\{X_{e_i}\}}_{i=1}^{t-1},w_e,L)\\ & \propto P(T_e|w_e,L)\times \sum _{s_{e_{t-1}}}P(X_{e_{t-1}}|s_{e_{t-1}})P(s_{e_{t-1}}|T_e)\times \cdots \\ & \times \sum _{s_{e_1}}P(X_{e_1}|s_{e_1})P(s_{e_1}|T_e)\\ & =P(T_e|{\{X_{e_i}\}}_{i=1}^{t-2},w_e,L)\sum _{s_{e_{t-1}}}P(X_{e_{t-1}}|s_{e_{t-1}})P(s_{e_{t-1}}|T_e)\end{array}$$

$$\begin{array}{rl}& P(T_e|{\{X_{e_i}\}}_{i=1}^{t-1},w_e,L)\\ & =\sum _{\{s_{e_i}{\}}_{i=1}^{t-1}}P(T_e,s_{e_i}|{\{X_{e_i}\}}_{i=1}^{t-1},w_e,L)\\ & \propto \sum _{\{s_{e_i}{\}}_{i=1}^{t-1}}P(T_e,s_{e_i},{\{X_{e_i}\}}_{i=1}^{t-1}|w_e,L)\\ & =\sum _{\{s_{e_i}{\}}_{i=1}^{t-1}}P(T_e,s_{e_i},{\{X_{e_i}\}}_{i=1}^{t-1}|T_e)\times P(T_e|w_e,L)\\ & =P(T_e|w_e,L)\times \sum _{\{s_{e_i}{\}}_{i=1}^{t-1}}[P({\{X_{e_t}\}}_{i=1}^{t-1}|s_{e_i})\times P(s_{e_i}|T_e)]\\ & =P(T_e|w_e,L)\times \sum _{\{s_{e_i}{\}}_{i=1}^{t-1}}\prod _{i=1}^{t-1}P(X_{e_i}|s_{e_i})P(s_{e_i}|T_e)\end{array}$$

$$\begin{array}{rl}& P(T_e|{\{X_{e_i}\}}_{i=1}^{t-1},w_e,L)\\ & \propto P(T_e|{\{X_{e_i}\}}_{i=1}^{t-2},w_e,L)\times \sum _{s_{e_{t-1}}}P(X_{e_{t-1}}|s_{e_{t-1}})P(s_{e_{t-1}}|T_e)\end{array}$$

Full text

I Introduction

BCIs promise to provide a new solution for people with major speech and muscle impairments to interact with their environment and communicate with others. EEG is a non-invasive method for recording brains electrical activity [1]. Being non-invasive, EEG-based BCIs have attracted significant attention, to be become a viable and safe solution.

P300 and Steady State Visually Evoked Potential (SSVEP) are more popular in BCI systems where the brevity of response time and the availability of several options are concerned. Researchers in the field have exploited these features to facilitate wheelchair navigation and develop spellers. In control and navigation applications SSVEP-based BCI systems are widely used due to their fast response. For instance, Muller et. al. proposed a four-command, SSVEP-based robotic wheelchair [2, 3]. In another study, P300 and SSVEP were used in a hybrid system to control a wheelchair [4]. However, EEG is known to have low signal to noise ratio. Studies have shown that fusion of other information sources (e.g. contextual information and/or history of user’s earlier selection) can enhance system performance [5, 6].

In applications with more than two command, usage of the P300 component might not provide the best performance. Typically in these BCIs not only the system requires multiple trials of each target in order to make a confident decision; but also, the system needs to map many options onto two classes which can lead to significant delay in decision making. Many researchers employ a different approach in which, the user navigates a cursor in a gridded space of options to the desired node and make a selection [7, 8, 9]. The Bremen BCI keyboard is an example of such systems in which, an optimized keyboard layout is presented to the user and they can perform letter by letter typing through navigating a cursor on a grid of characters [7, 8]. Similarly, Iwan et.al. [10] used Error-related potential (ErrP) as a biological signal to navigate the cursor through the gridded space of options to select the desired option. But most of these researches such as  [7, 8, 10], do not use neither the layout information or the confidence by which earlier selections are performed.

Our earlier work, Flashtype [11], is utilizing a probabilistic graphical model which uses context information and the evidence accumulated from earlier decisions to calculate prior probability distribution over the state space. In this study, we propose a new probabilistic graphical model maintaining the use of context information and the evidence accumulated from earlier decisions. The introduced statistical inference mechanism provides a new Selection criterion to estimate user intent. We report the performance of these models using our earlier designed BCI, Flashtype™ [11]. A simulation study using real code modulated VEP (c-VEP) data collected during a Calibration session from seven subjects is used to compare possible performance improvements using the proposed methods.

II Methods

II-A BCI Graph Navigation Framework

In this section a general framework for a BCI navigation system is described. As shown in Fig. 1, a navigation graph contains a set of connected nodes. At each node an immediate action of moving to a neighbor node (from $m$ possible neighbor nodes in a connected graph with $n$ nodes) or current node selection can be taken. Therefore, a set of immediate actions are needed to converge to the target node through a specific path. In a BCI navigation system, each immediate action of moving to a neighbor node or current node selection is taken by using subject’s brain evidence and contextual information. In our system, each navigation sequence which is called an epoch contains a set of immediate actions through $t$ trials ($1\leq t\leq N_{t}$) for converging to the target node. At the end of each epoch, navigation sequence to the target node is finalized by a selection, based on a Selection criterion. In this probabilistic framework, in each trial next neighbor node is decided by estimation of subjects intended action. In our system, this probabilistic estimation in epoch $e$ and trial $t$ is attained by fusion of (1) EEG evidence, (2) prior probabilities of each node being target and (3) history of previous actions and their probabilities in that epoch.

II-B Graphical Models for Statistical Inference

In this section we introduce two probabilistic graphical models used for navigation and inference. In both models, prior probabilities over the nodes are recursively updated as the user is navigating throughout the graph in each trial. The goal, is to estimate the next command or immediate action, $s_{e_{t}}$, at epoch $e$ and trial $t$. Here the context information, $\omega_{e}$, defines a prior distribution over the space of nodes. Moreover, $X_{e_{t}}$ is the EEG evidence corresponding to $s_{e_{t}}$, and $L$ represents the layout of graph node. In graphical model demonstrated in Fig. 2 (a), $T_{e}$ represents the true state of the system in epoch $e$. Fig. 2 (b) shows the graphical model introduced in our previous related work [11] in which $y_{e_{t}}$ is the desired pointer location at trial $t$ of epoch $e$; and $A$ represents a particular action assignment on the graph.

Two different approaches are exploited for estimation of the next command, $s_{e_{t}}$, in each graphical model. According to our earlier work in [11], we have the following proportion for graphical model (b).

[TABLE]

In graphical model (a) the joint probability of action decision and true state of the system given the observed EEG evidence and context information is maximized. Assume that $\{\mathbf{X}_{e_{i}}\}_{i=1}^{t},\mathbf{X}_{e_{t}},w_{e}$ and $L$ are observed. Therefore, the joint probability of action decision and true state of the system given all observed variables can be written as,

[TABLE]

In equation (2), $P(T_{e}|{\{X_{e_{i}}\}}_{i=1}^{t-1},w_{e},L)$ represents the probability over the state space estimated from the context and observed EEG in earlier trials. This probability mass function can be updated after each trial as shown in (3).

[TABLE]

For more detail on derivations for Equation 3 please see Appendix APPENDIX. In the (3),

• •

$P(\mathbf{X}_{e_{t}}|s_{e_{t}})$, represents likelihood of EEG evidence given each possible class of actions in trial $t$ of epoch $e$.

• •

$P(s_{e_{t}}|T_{e})$, is the probability of taking each action class given true state of the graph in trial $t$ of epoch $e$.

II-C Target Selection Criteria

Two decision criteria for epoch conclusion was utilized; first the user need to choose a “Select” command, second, if the ratio of the current pointer location probability, to the next most probable location exceeds a predefined threshold ($\frac{P(T_{e}=currentLocation|{\{X_{e_{i}}\}}_{i=1}^{t-1},w_{e},L)}{P(T_{e}=SecondMostProbableLoc|{\{X_{e_{i}}\}}_{i=1}^{t-1},w_{e},L)}>Tr$) the system selects current node. In this manuscript we refer to this condition as Probabilistic Selection Criterion (PSC).

III Simulation Results

III-A Simulation Study Design

In this study, a code visually evoked potential (c-VEP) based BCI gridded keyboard, FlashType, [11] is utilized to assess the effectiveness of; (1) graphical models described in Section II-B and (2) target Selection criteria explained in Section II-C. Here each character on the keyboard is represented by a node of degree four (m=4) on a graph with 35 nodes. A 6-gram language model provides prior context information for each character111Note that the word suggestion feature of the FlashType [11] was turned off during this simulation.. Typing each character consist of a sequence of cursor movements or actions in an epoch finalized by selecting a target character. Four checkerboards located at the corners of the gridded keyboard, are used as stimuli. Each checkerboard is mapped to a different action moving the cursor on the gridded keyboard.

A Monte Carlo based simulation is performed over typing ten words from individual sentences, called a session, each with 30 runs. The words come from five different difficulty levels based on language model prediction. Levels with higher difficulty imply lower prediction probability based on language model and vice versa. Typing the following ten words ”the, and, with, will, seat, between, seen, please, buys, makeup” is defined at a simulation run. The level of difficulty increases by one after every two words.

Seven pre-recorded c-VEP calibration data sets with high, average, and low classification accuracies have been used to produce realistic EEG features for simulation [12, 13]. The accuracy of each data set has been estimated by employing a leave-one-out cross validation among different selection options.

To simulate typing each character, the system assumes subject’s intended navigation sequence in each epoch is equal to the path with minimum number of cursor movements toward the desired character. Therefore, in each trial true action command is chosen based on the determined minimum path to the desired character. Next, samples from true action command class are drown based on estimated class conditional distributions. The set of likelihood scores for i.i.d drawn samples are then calculated using the class conditional densities.

Three probabilistic models for estimating subject’s next intended direction or command are evaluated and compared. In graphical model (a) joint probability of true state and actions $P(s_{e_{t}},T_{e}|{\{X_{e_{i}}\}}_{i=1}^{t-1},X_{e_{t}},w_{e},L)$, in graphical model (b) posterior probability of states, $P(s_{e_{t}}|{\{X_{e_{i}}\}}_{i=1}^{t-1},X_{e_{t}},w_{e},L)$, and in the case of using no graphical model the likelihood probability of states, $P(\mathbf{X}_{e_{t}}|s_{e_{t}})$, are calculated. The calculated probabilities in each case need to reach a certain Confidence Threshold, for the corresponding command to be chosen. Otherwise, another set of likelihood scores is randomly drawn from the conditional densities and conditional probabilities for each model are calculated again. This procedure continues until the Confidence Threshold or maximum allowed number of repetitions (5 in this simulation) has been reached.

III-B Model Comparison Results

Two PGMs (probabilistic graphical models) explained in Section II-B and two target Selection criteria explained in Section II-C are compared in this section. A PGM along with a target Selection criterion can be used for each typing approach. Different pairs of PGMs and Selection criterion are used for comparision.

Using the simulation study explained in section III-A, effect of employing each approach on average typing duration of each session is illustrated in Fig. 3. For each pre-recorded calibration data sets, total time for completing each session (typing ten words) was first calculated and then averaged over all 30 simulated runs. Note that each likelihood score in simulation, will require 1.05 second of EEG observation in a real experiment scenario. This figure indicates using PGMs enhances the typing speed specifically for the users with low calibration performance. Time needed to complete a session was significantly decrease when either graphical model (a) or (b) was used compared to not using a graphical model, the significant difference is calculated based on a t-test with $\alpha$ threshold of 0.001 ($p<0.001$). The significant decrease is observed in data sets with calibration accuracies less than 90%. Overall, adding the PGM (b) along with PSC provides the highest typing speed improvement.

The difference between graphical model (a) and (b) can be seen in equation 1 and 2. While graphical model (b) is marginalizing out the desired pointer location over nodes, graphical model (a) chooses the maximum joint probability of the true state of nodes and immediate actions. Hence, the context information has a stronger effect on graphical model (a). Figure 4 emphasises this difference by showing the time needed to type two complete words, when context information is in favor of user intend (upper plot) and opposing the user intend (lower plot). When context is in favor of user intend graphical model (a) performs better. On the other hand, when the context is not in favor of user intend graphical model (b) performs better.

IV Discussion

In this study, we introduced two PGMs which use context information and previously observed EEGs in addition to the EEG recorded during the current trial. Our simulation results show PGMs along with PSC can significantly enhance typing speed especially for users with poor EEG classification performance.

Similarly, these models can be employed in SSVEP based wheelchair navigation. Fig. 5 shows the example plan of a home environment as a set of user presence. Each location on this plan has a specific user presence probability. Assuming four SSVEP options (turning left, turning right, forward, and backward) to move the wheelchair, context information for each of these discrete options is extracted based on the spatial integration of possible targets in each direction.

APPENDIX

According to Fig. 2.a the probability of the true state given observed random variables can be written as:

[TABLE]

Equation 4 can be evaluated in recursive mode.

[TABLE]

Which is equal to equation 3.