An online brain–machine interface using decoding of movement direction from the human electrocorticogram

Five subjects (S1–S5) suffering from intractable pharmaco-resistant epilepsy (table 1) voluntarily participated in the study after having given their informed consent. The study was approved by the Freiburg University Ethics Committee.

For pre-neurosurgical epilepsy diagnostics, the subjects were implanted with an 8 × 8 grid of subdural surface electrodes (Ad-Tech, Corp., 1 cm inter-electrode distance, 4 mm electrode diameter) covering parts of the primary and pre-motor cortex (figure 1). S1, S2, S3 and S4 had additional ECoG stripes implanted. In addition to the signal from the subdural electrodes, 21 surface EEG channels, one or two bipolar electro-oculography (EOG) channels, the electrocardiography (ECG) channel and several electromyography (EMG) channels were recorded. Signals from the ECoG electrode stripes, EEG, ECG and EMG channels were not analyzed in this study.

Zoom In Zoom Out Reset image size

Recordings from all electrodes were digitized at 1024 Hz sampling rate for S1, S2 and S3 (Brainbox EEG-1164 amplifier, Braintronics B. V., Almere, Netherlands) and at 2500 Hz sampling rate for S4 and S5 (AC441-01 Neuvo amplifier, Compumedics Limited, Abotsford, Australia). Recordings for S1, S2 and S3 were made using a hardware high-pass filter with 0.032 Hz cutoff frequency. Recordings for S4 and S5 were made without the hardware high-pass filter. Experiment control and paradigm presentation were performed using our own laboratory software. Subsequent data analysis was performed using MATLAB (MATLAB versions 7.4-7.11, Natick, Massachusetts: The MathWorks Inc., 2007–2011).

Subjects interacted with an experimental paradigm shown on a computer screen (figure 2). The experiment was carried out in sessions, defined as uninterrupted time epochs in which subjects continuously interacted with the paradigm. Each session consisted of 50 or 25 trials after which the subject stopped performing the task. Each trial consisted of a pause phase (1–2 s, random, uniformly distributed) followed by a preparatory informative cue presentation (displayed for 1–2 s, random, uniformly distributed), which informed the subject to prepare for moving a joystick to the left (purple rhomboid) or to the right (red rhomboid) using the hand contra-lateral to the implantation site. After a delay of 2–3 s (random, uniformly distributed), a go cue (green dot) was presented and subjects had to initiate the movement during the next 1 s. During the movement, no visual feedback was given to the subject. After reaching the joystick end position, they had to keep it in this position for additional 2 s. If subjects did not follow this sequence correctly, the trial was stopped and not used. This was done to ensure stereotypical movements of the subjects after the go cue and no movements before the go cue. Subsequently, the cursor on the screen moved in the direction in which subjects moved the joystick (feedback phase).

Zoom In Zoom Out Reset image size

Each experiment started with an introduction session in which the subject was familiarized with the task. Subjects were encouraged to perform correct as well as incorrect trials, so that they would get used to the task and to the error messages written on the screen. Once the subject was familiar with the task, the introduction session was stopped and one or two training sessions followed. During the feedback phase of the training sessions, the cursor always moved over to the maximum distance from the centre in the correct direction. Once the training sessions were finished, a model was built from the collected data. One or more brain-control sessions followed, identical to the training sessions, except for the feedback phase. Now, the distance travelled by the cursor was proportional to the posterior probability for the cued movement direction as decoded with the trained model. A vertical line was positioned at the half of the maximum distance the cursor could travel, signifying posterior probability of 0.5. Every time the posterior probability was higher than 0.5, the cursor would cross the line and the word 'Tor' (German for 'goal') was written on the screen, providing positive feedback to the subject. After every brain-control session the model was re-built using one or more of the previous sessions as training data, sometimes with a change in the selection of electrodes. Table 2 summarizes the experiments for all subjects.

Data from relevant (table 2) sessions was common average referenced using recordings from all electrodes on the 8×8 ECoG grid that did not have any defects or did not record epileptic activity.

First we chose the strategy for selecting the electrodes to be used in model building. We either selected all electrodes that showed a hand or arm motor response during the electrical stimulation mapping (ESM; ESM H&A motor electrodes; Foerster 1931, Uematsu et al 1992) or we chose the electrodes by visual inspection of the recorded ECoG amplitudes for each movement direction (table 2).

Our decoding model was based on several features of low-pass filtered ECoG signals (second-order symetric Savitzky–Golay filter with window length optimized between 0.25 and 1 s; Savitzky and Golay 1964, Steinier et al 1972). Following filtering, features were taken at different time points with respect to the go cue. Model building consisted of testing different values of parameters defining the feature selection and the decoder used for classification. Feature selection consisted of selecting: (i) the window length of the Savitzky–Golay filter, (ii) the number of features from one electrode, (iii) the time of the first feature relative to the go cue, and (iv) the temporal distance between the first and the last feature. For decoding, we used regularized linear discriminant analysis (RLDA; Friedman 1989), which uses an additional parameter, the regularization coefficient (v), to improve generalization. ECoG recordings for S4 and S5 were not high-pass filtered by the recording system. Therefore, to remove low frequency potential drifts for S4 and S5, we subtracted the low-pass filtered ECoG signal that was obtained by filtering using a causal running average filter. In addition to other model parameters, for S4 and S5, we also selected between different values of (vi) the window length of the running average filter. For each of these parameters, we defined a set of values and tested every combination of these values using five-fold cross validation. The set of parameter values which gave the highest estimated normalized decoding accuracy (DA) was used to build the model on the entire set of training data:

$$\begin{equation} DA = \frac{1}{{N_{{\rm class}} }}\sum\limits_{i = 1}^{N_{{\rm class}} } {\frac{{c_i }}{{n_i }}} \end{equation} \tag{ 1 }$$

where N_class is the number of classes (in our case 2; left and right movements), c_i is the number of correct trials for a given class and n_i is the total number of trials for a given class.

During the online experiment, the time for model building was limited. Therefore, the number of tested parameter values was reduced. In the offline analysis we always used a more exhaustive range of parameter values: (i) Savitzky–Golay filter window length: ¼, ½, ¾ and 1 s; (ii) number of features from one electrode: 1, 2, 3, 4 and 6; (iii) temporal distance of the first feature relative to the go cue: from 0 till 1.4 s after the go cue in steps of 0.1 s; (iv) time interval between the first and the last feature: $\frac{1}{16}$, $\frac{1}{8}$, $\frac{1}{4}$, $\frac{3}{8}$, $\frac{1}{2}$, $\frac{3}{4}$ and 1 s; (v) regularization parameter: 0, 0.001, 0.1, 0.3, 0.5, 0.7, 0.9 and 0.99; (vi) window length of the running average filter: 5, 10 and 20 s (for S4 and S5 only).

To optimize decoding, proper electrode selection is important (Muller et al 2000, Lal et al 2004, Demirer et al 2009). In the case of patients recovering after intracranial surgery, an additional restriction is that the model has to be built in the short amount of time available for experiments and, hence, needs to be based on only a small amount of training data. This restricts the number of possible electrode selections that can be tested. For this reason, before the brain-control sessions, we selected the electrodes used for model building based either on the results of ESM or on visual inspection of the neural responses, thereby not optimizing the electrode selection on the basis of DA.

Offline analysis was used to test whether selecting the electrodes in a different manner could have increased the DA. We restricted the electrode selection to the ESM H&A motor electrodes, trying to avoid electrodes where information could be a result of some kind of artefact (e.g. eye movements) and minimizing the influence of sensory feedback induced by the subjects' movements. To further minimize the influence of sensory input, we also considered the subset of the ESM H&A motor electrodes that lay over the motor cortex according to the sulci reconstruction (ESM H&A motor + SR motor electrodes).

Thus, two electrode sets were considered for the offline electrode selection strategy: (a) all ESM H&A motor electrodes and (b) ESM H&A motor + SR motor electrodes. For each of these sets, we considered (i) single electrodes belonging to the electrode set (a or b), (ii) electrode pairs (vertical or horizontal) within the electrode set, (iii) three neighbouring electrodes within the electrode set, (iv) four neighbouring electrodes within the electrode set and (v) all electrodes together belonging to the electrode set. We evaluated the DA of all electrode subsets for all parameter values (see above) by five-fold cross validation on the training data and selected the parameter values and electrode subset yielding the highest DA for (i), (ii), (iii), (iv) and (v) separately. These subsets and parameter values were then used to decode the brain-control sessions offline. Sessions used for training and testing the model were chosen in the same way as in the online experiment (table 2).

To confirm that neural responses to left and right movements were significantly different from each other and from baseline activity, a Mann–Whitney–Wilcoxon test was applied between neural responses at every time point. Neural activity recordings were first low pass filtered with the Savitzky–Golay filter (symmetric, second-order, 0.5 s window length). Since the response had to be present within a limited time after the trigger, we tested the epoch of the neural activity from the go cue until 2 s after the go cue. Baseline neural activity was defined as all neural activity outside the epochs described above. To sample the baseline activity distribution properly, we removed the autocorrelation of the low-pass filtered activity arising from the filtering procedure by sampling the baseline activity only every 0.5 s. To reduce the computation time of the significance testing, we tested significance of the neural responses every 31.25 ms (corresponding to 32 Hz), instead of testing for every recorded time point.

Due to the large number of statistical tests, correction for multiple testing was necessary to control the number of falsely rejected null hypotheses. We used the Benjamini–Hochberg procedure (Benjamini and Hochberg 1995) with a correction for dependent statistics (Benjamini and Yekutieli 2001) to set the false discovery rate for one subject at the level of 5% for all tests. A neural response to the go cue on a certain electrode was considered significant if there was at least one time point for which the neural response to the go cue for right movements was significantly different from baseline neural activity or, alternatively, at least one time point for which the neural response to the go cue for left movements was significantly different from baseline neural activity. This condition needed to be satisfied after the correction for multiple testing was applied, taking into account all time-point-wise tests for both conditions. A neural response to the go cue was considered significantly different between left and right movements if there was at least one time point for which the neural response to the go cue for right movements was significantly different from the neural response to the go cue for left movements, after the correction for the multiple testing was made.

Since the neural signal epochs used for decoding were aligned to the go cue, differences in movement reaction times and, hence, in the onset times of neural responses time-locked to movement onset, could have quite different effects on DA, depending on whether they originate from trials of the same movement type (either left or right) or whether they stem from a systematic difference in reaction times for different movement types (left versus right). In the latter case, the systematic difference in response onset by itself would allow for correct directional decoding, even for otherwise identical neural responses. Removing the systematic reaction time difference by aligning the neural signal epoch on movement onset instead of on go cue should then reduce the DA. On the other hand, differences in reaction times for trials of the same movement type would increase the variability of the neural responses of both movement types when aligned to the go cue and, hence, lead to a reduction in DA. Removing the effect of reaction time variability by aligning the signal epochs on movement onset, instead of on go cue, would then be expected to result in an increase of DA.

To test which of these two scenarios were true, we realigned the neural responses to movement onset, thereby removing, or at least reducing, both response onset variability and systematic response onset latencies due to differences in reaction times. We ran the whole experiment offline as if it was an online experiment, i.e. using the same sessions for training and testing (table 2).

Electrical cortical stimulation through the electrode grid was performed using an INOMED NS 60 stimulator (INOMED, Germany) as a part of the clinical procedure. Stimulation trains of 7 s duration consisted of 50 Hz pulses of alternating polarity square waves of 200 µs each. The intensity of stimulation was gradually increased up to 15 mA or to the induction of sensory and/or motor phenomena. Subjects were unaware of the timing of stimulation, unless these phenomena occurred. Phenomena were reported by the subject, specifying the limb of origin and the type of sensation (motor or somatosensory). Consequently, ESM maps were created and used to select the electrodes used for online brain control (figure 3).

Zoom In Zoom Out Reset image size

Data from post-implantation MRIs were used to reconstruct the positions of the Sylvian fissure, central sulcus, postcentral sulcus and precentral sulcus with respect to the electrodes (figures 1 and 3). The reconstruction informed us whether an electrode was lying over one of the reconstructed sulci or not. We defined the sulci reconstruction (SR) motor electrodes as set of electrodes located over the central sulcus, over the precentral sulcus, or in-between the central and the precentral sulci according to the sulci reconstruction. Sulci reconstruction was available only after the online experiment had finished.

We found significant neural responses to go cue (figure 4) on a large number of electrodes (47 out of 64 for S1, 37 out of 64 for S2, 52 out of 64 for S3, 19 out of 64 for S4 and 44 out of 64 for S5; 62% of the electrodes on average). In contrast, only a small number of electrodes showed significant differences between neural responses to left and right movements (8 electrodes for S1, 1 for S2, 7 for S3 and none for S4 and S5, 5% of the electrodes per subject on average). These results show that, even though the neural response to hand movements are widely distributed, only a small fraction of the responses contained directional information.

Zoom In Zoom Out Reset image size

During the online experiment, we ran two training and three brain-control sessions with subject S1, one training and three brain-control sessions with S2, one training and two brain-control sessions with S3 and one training and one brain-control session with S4 and S5 each (table 2). Table 3 shows the overview of the chosen parameter values during the model building phase for each of the brain-control sessions.

We achieved significant (p < 0.01, binomial test) directional decoding in online closed loop hand movement direction decoding in four out of five subjects and eight out of ten sessions (figure 5(A), table 4) with average DA of 75%. We used two different strategies to select the electrodes used for model building. For the first brain-control session for S1 and all brain-control sessions for S2, S4 and S5 we used all ESM H&A motor electrodes (average DA 71%). For the other brain-control sessions, we selected the electrodes which showed strongest tuning during training, not necessarily restricting ourselves to ESM H&A motor electrodes (average DA 80%). The difference in DA between these two strategies was significant (p < 0.05, Fisher's exact test). On the other hand, we found no significant difference (p = 0.48, Fisher's exact test) between strategies of using only the last preceding session to train the model (average DA 77%) and using multiple sessions (average DA 74%).

Zoom In Zoom Out Reset image size

In the second brain-control session of S3, we used neural activity measured with only two neighbouring electrodes, 1 cm apart, for decoding (figure 5(A), electrodes marked in figure 3). The DA of 76% achieved by using only these two electrodes was not significantly different from the DA in the remaining sessions (mean DA = 75%, p = 1, Fisher's exact test).

We evaluated the DA from different electrode selection strategies and different electrode subsets (figure 5(B)). For ESM H&A motor + SR motor electrodes, the DA from four neighbouring electrodes (average DA 78%) was not significantly different from the DA from all subset electrodes (average DA 76%; p = 0.61; Fisher's exact test) or the DA from the brain-control sessions (average DA 76%; p = 0.56; Fisher's exact test). This shows that most of the directional information was already present in the signals recorded from small subsets of four neighbouring electrodes (contained within a 14 mm × 14 mm area). Additionally, ESM provides a strong indication that all four of these electrodes were recording neuronal signals from the hand and arm area of the motor cortex.

Using all ESM H&A motor electrodes yielded the highest DA (average DA 81%), with a tendency to be higher than the DA from brain-control sessions (p = 0.07; Fisher's exact test) and the DA from using only all ESM H&A motor + SR motor electrodes (p = 0.07; Fisher's exact test). Note that the 'all ESM H&A motor electrode set' can include electrodes that lay over pre-motor and somatosensory cortex according to the sulci reconstruction (see section 2.6).

Figure 6 shows the spatial distribution of DA over the electrode grid when electrode quartets were used for decoding. For all subjects, the maxima of the DA had at least one electrode in the quartet belonging to the ESM H&A motor electrode set, with other high DA quartets grouped around the locations of the maxima. This is expected for a task involving hand and arm movements and confirms that the neural responses used for decoding were not the product of artefacts.

Zoom In Zoom Out Reset image size

Inspection of the joystick movements (figure 7(A)) revealed that, apart from reaction time and movement direction, rightward and leftward movements were quite similar, both across movement types (left versus right) and across trials within the same movement type (either left or right). Following the go cue, subjects initiated the movements (defined as crossing 15% of the maximum joystick deflection in horizontal direction) after 415 ± 87 ms (S1 304 ± 12 ms; S2 156 ± 12 ms; S3 439 ± 13 ms; S4 661 ± 33 ms; S5 513 ± 16 ms). We observed a significant difference in reaction time between left and right movements (p < 0.05; Mann–Whitney–Wilcoxon test) for some sessions of S2 and S4 (figure 7(B)).

Zoom In Zoom Out Reset image size

To test whether response onset variability has an effect on the DA, we re-aligned the neural responses to movement onset and re-ran the experiment using the same sessions for training and testing (table 2). We found that, in all subjects, the resulting DA was higher after alignment to movement onset than during the brain-control sessions where trials were aligned on the go cue (figure 7(C); DA increase 0.08 ± 0.03%, p < 0.05; signed Mann–Whitney–Wilcoxon test). This indicates that the selected neural responses were indeed coding for movement direction and that correct decoding was not due to systematic response latency differences for different movement types.