The Berlin Brain–Computer Interface: Non-Medical Uses of BCI Technology

Brain–computer interfacing (BCI) is a steadily growing area of research. While initially BCI research was focused on applications for paralyzed patients, increasingly more alternative applications in healthy human subjects are proposed and investigated. In particular, monitoring of mental states and decoding of covert user states have seen a strong rise of interest. Here, we present some examples of such novel applications which provide evidence for the promising potential of BCI technology for non-medical uses. Furthermore, we discuss distinct methodological improvements required to bring non-medical applications of BCI technology to a diversity of layperson target groups, e.g., ease of use, minimal training, general usability, short control latencies.

1 Introduction

Brain–computer interfacing (BCI), i.e., the ability to transfer and use information from distinct brain states for communicating with a machine has in the past years received considerable attention (Wolpaw et al., 2002; Birbaumer, 2006; Allison et al., 2007; Dornhege et al., 2007a; Schalk, 2008; Krusienski and Wolpaw, 2009). While the mainstream of research addressed improvements of paradigms (Hill et al., 2005; Citi et al., 2008; McFarland et al., 2008; Hwang et al., 2009; Williamson et al., 2009; Höhne et al., 2010; Schaefer et al., 2010; Schreuder et al., 2010; Treder and Blankertz, 2010) and data analysis technology (Parra et al., 2005, 2008; Dornhege et al., 2007b; Lotte et al., 2007; Blankertz et al., 2008a; Tomioka and Müller, 2010) as to allow for an easier, more convenient and faster use of BCI, this was mainly done for communication purposes with the overall aim to help paralyzed patients (Kübler et al., 1999, 2000, 2001, 2005; Birbaumer et al., 2000, 2003; Neuper et al., 2003; Pfurtscheller et al., 2003; Birbaumer and Cohen, 2007; Dobkin, 2007; Müller-Putz et al., 2007; Cincotti et al., 2008; Daly and Wolpaw, 2008; Conradi et al., 2009). Recently BCI technology has also been used for a larger audience, namely for non-medical purposes. Here, not only communication is central, but BCI technology has gained popularity in the form of measurement devices, that allow to access respectively decode macroscopic brain states such as attention, performance capability, emotion etc., in real-time (Dornhege et al., 2007a; Müller et al., 2008). The signals extracted by BCI techniques are then employed to exploit this novel information for improved man–machine interaction. This allows to optimize and to enhance human performance and to achieve potentially novel types of skills.

This paper discusses these recent non-medical developments, with a focus on the work of the authors, and puts them into perspective. Clearly, a wider use of BCI technology has become possible only through the use of modern machine learning and signal processing methods, that allowed to relocate the burden of training from a learning subject toward statistical learning machines and thereby achieve BCI communication for a naïve user already in the first session (Blankertz et al., 2002, 2007a, 2008b). This issue as well as technological requirements of non-medical BCI use are discussed in Section 2. In Section 3, we outline how the real-time decoding of mental states, such as fatigue or workload, can be used to optimize an operator’s performance. Then, BCI for multimedia applications and gaming, i.e., for a novel type of user performance is discussed in Section 4, and finally we briefly conclude with some remarks on the future of man–machine interaction.

2 Improvements in BCI Technology

A broader applicability of BCI technology for alternative uses requires additional methodological steps that have not been in the focus of BCI research aiming at patients’ applications. Here, we discuss improvements in ease of use as well as broad and robust applicability: short preparation: Section 2.1; minimal user training: Section 2.2; minimal calibration of the system: Section 2.3; applicability for a broad range of users: Section 2.4; practicing short latency BCI operation: Section 2.5; BCI control that takes into account the current state of the user: Section 2.6.

2.1 Dry Electrodes

Wet electrodes are very time-consuming to setup. This is one main reason why EEG technology is not adopted easily by a wider audience. Early prototypes of dry electrode technology were developed in the late 1960s and early 1970s (Richardson et al., 1968; Bergey et al., 1971) and since then various dry and insulating electrode materials have been tested (Searle and Kirkup, 2000) and also a capacitive electrode coupling approach has been achieved (Oehler et al., 2008). Since this technology represents an easy-to-use alternative to classical wet electrodes, dry electrodes are still a sought-after solution today. Besides the shortened setup time there are also other benefits for long-term monitoring. Electrodes, dependent on gel could dry up, while dry electrodes can stay functional. For example, long-term ECG measurements for patient monitoring could indicate potential cardio-vascular problems for patients in high risk groups and various attempts have been made to increase practicability by combining it with wireless technology (Catrysse et al., 2004; Coosemans et al., 2006), flexible electrodes (Hoffmann and Ruff, 2007; Baek et al., 2008) and by including such devices into wearable textiles (Muhlsteff and Such, 2004; Carpi and Rossi, 2005).

A recent development (Gargiulo et al., 2010) uses little dry sensors made of silicone conductive rubber that are attached to the scalp using a skin compatible super glue. Although BCI paradigms have been used, data is not evaluated in the sense of BCI performance but rather as correlation coefficient between signals recorded from the novel dry sensors and concurrently acquired standard EEG. At the optimal delay of 50 samples, a mean correlation coefficient of 0.76 was found.

A different development was evaluated in Sellers et al. (2009) in the context of the matrix speller (Farwell and Donchin, 1988). EEG signals were acquired concurrently with a novel hybrid dry electrode sensor array (HESA) and conventional wet electrodes (Cz, Pz, PO7, PO8 from each system with a spatial distance of about 4 cm). Data recorded from both type of sensors during a standard spelling task was classified offline using the same algorithm. Performance was comparable for both systems (mean accuracy across eight participants was 67.5 vs. 70.5% for dry vs. wet sensors).

Two dry sensor systems have been evaluated with respect to an SSVEP (Morgan et al., 1996; Middendorf et al., 2000; Cheng et al., 2002; Allison et al., 2008) paradigm: Luo and Sullivan (2010) used a single-channel dry sensor in a four-class setting. An average detection rate of 75.8% was obtained in offline analysis for the best parameter setting with a very high variability between participants ranging from 5 to 100%. A helmet with 28 capacitive sensors is used in (Oehler et al., 2008) for a two-class SSVEP setting. The mean detection accuracy across four participants was 81% for the classification of 7 s long-time windows. Both report about online BCI operation restricted to one participant.

The only systematic study to date in which dry sensor technology was used to provide online BCI feedback is (Popescu et al., 2007). Here, a prototype solution with six dry electrodes was evaluated with respect to motor imagery driven BCIs. The electrodes were gold-coated and had a square shape of 0.5 × 0.5 cm² with multiple pins attached. Various springs and joints were necessary to ensure a comfortable fit. A classic BCI 1D feedback paradigm was tested and compared to the results of a cap with 64 wet electrodes. While the information rate was approx. 20% lower for the dry electrodes, peak information rates of 36 bits/min were reached on occasion, which is on par with state-of-the-art gel-based BCI performance. Note that the use of only six electrodes would easily allow to run the new system with a tiny EEG amplifier and a pocket PC.

2.2 Minimal User Training

The first approach to establish a pure BCI communication channel, which does not rely on any neuromuscular output pathways was described in Birbaumer et al. (1999) and Elbert et al. (1980). In this operant conditioning variant of BCI, the subject has to learn the self-control of slow cortical potentials at central scalp position (Elbert et al., 1980; Rockstroh et al., 1984; Birbaumer et al., 2000), which requires intensive training on the side of the user. Later, an approach was introduced relying on voluntary modulation of sensorimotor rhythms (SMR; Neuper et al., 1999; Pfurtscheller and da Silva, 1999; Wolpaw et al., 2000; Krusienski and Wolpaw, 2009), which substantially reduced the required training time (Vaughan et al., 2006). But still user training in the order of several sessions was necessary in most cases due to the relatively fixed way of feature extraction which does not completely account for the high inter-personal variability with respect to the brain signatures of natural (i.e., untrained) control commands. BCI systems that are based on the detection of potentials that are related to external stimuli (Wolpaw et al., 2002) typically require less user training. In the following we only focus on system which use endogenously altered mental states.

Machine learning based BCIs use EEG features of larger complexity that can be fitted better to the individual characteristics of brain patterns of each user. To this end, there is often an initial calibration period, in which signals are acquired while the user generates control commands according to cues without receiving feedback. Machine learning algorithms are applied to these labeled data to infer features (e.g., spatial filters, frequency bands) that are optimized for the BCI performance of the individual user (Blankertz et al., 2002, 2007a; Parra et al., 2003; Dornhege et al., 2007a). In Blankertz et al. (2008b) we investigated which proportion of naive subjects could successfully use an SMR-based system in the very first session. Participants of the study were 14 individuals who never performed in a BCI experiment before. For one subject, no distinguishable classes were identified from the calibration data. The other 13 subjects performed feedback: 1 near chance level, 3 with 70–80%, 6 with 80–90%, and 3 with 90–100% hits. The results of all feedback runs using the Berlin brain–computer interface (BBCI) are shown in Figure 1.

FIGURE 1

Figure 1. Left: Feedback accuracy of all runs (gray dots) and intra-subject averages (black crosses). Right: Histogram of accuracies obtained in BBCI-controlled cursor movement task in all feedback runs of the study.

Instead of an offline calibration, it is also possible to start with BCI feedback right from the beginning by using a subject-independent classifier which is then adapted to the individual trial-by-trial. Although some research groups claim that it is possible to do this in an unsupervised manner (Li and Guan, 2006; Blumberg et al., 2007), all online studies published so far use supervised adaptation method for the initial period (Vidaurre et al., 2006, 2007; Wang et al., 2007; Vidaurre and Blankertz, 2010). The term “supervised” means that the system needs to know the true intention of the subject for adaptation, which is typically done by cueing the subject to generate certain control commands. The true application that can be controlled by the user can start only after this adaptive calibration.

Relying on an adaptive calibration users can be led efficiently and fast to a successful BCI control within their first session. As an illustrative example, Figure 2 shows the feedback performance of a naive subject within her very first session from the very first trial on (using the technique presented in Section 2.4). Dots indicate the average performance of 20 trials. Runs of adaptive calibration are show in magenta and orange color. After only 20 trials, performance is at 85%, and after 60 trials performance is almost perfect for the rest of the session. More details about the method can be found in Section 2.4. These studies show that a machine learning based approach to BCI is able to let BCI novices perform well from the first session on. Still, we would like to remark that there is a non-negligible portion of users, for which this quick-start approach is not successful, see Section 2.4.

FIGURE 2

Figure 2. The graph shows the feedback performance of one BCI-naive subject from the very first trial on. Results are from one single session in which 8 runs of 100 trials (about 15 min) each have been recorded. There was no calibration period before. Feedback started with a general, subject-independent classifier which was adapted trial-by-trial. Dots indicate the average feedback performance (1D cursor control) of 20 trials. The mean performances of each run of 100 trials is shown as bars. The three colors relate to three different processing methods, which are explained in Section 2.4.

2.3 Minimal Calibration: Subject-Independent Classifiers

Even when user training is avoided by taking a machine learning approach to BCI, see Section 2.2, there remains a reduced but still time-consuming preparatory step: the calibration of the system to the user’s characteristic activation pattern. Lately, a number of attempts have been made to overcome the calibration by means of specifically developed machine learning techniques, e.g., Kaper and Ritter (2004), Lu et al. (2009) for P300 and Allison et al. (2008), Cecotti (2010) for SSVEP as well as Krauledat et al. (2008) for SMR-based BCIs where feedback and calibration data of multiple sessions of BCI-experienced subjects were employed to identify subject-dependent prototypical spatial filters. These spatial filters were shown to provide better generalization properties compared to single-session filters and could therefore be used in following sessions without the need to recalibrate the system for these subjects. The offline results were confirmed by online experiments and no loss of classification performance was observed. However, still a fairly large number of sessions of the same subject were required. More recently, a subject-independent zero-training SMR-based BCI system was developed by harvesting a large library of previous BCI experiments (Fazli et al., 2009). This large library allowed to combine a very large set of subject-dependent classifiers into a single subject-independent classifier by choosing appropriate weights for creating a very sparse set of voting classifiers. Among several tested methods (such as k-nearest neighbor (kNN), support vector machines (SVMs), linear discriminant analysis (LDA) and others) l₁ regularized regression performed best. The resulting subject-independent classifier performed almost as well as standard, state-of-the-art subject-calibrated methods across the data from 91 participants.

A different approach is presented in Lotte et al. (2009). In the offline evaluation which is limited to data sets of nine subjects, the best subject-independent classifier obtains on average 10% less in accuracy compared to the best subject-specific classifier.

2.4 BCI Deficiency and Countermeasures

One of the major obstacles before BCI technology can be widely applied in non-medical fields is the problem of “BCI Deficiency,” meaning that for a non-negligible portion of users, BCI systems cannot detect their intentions accurate enough to let them control applications. Gaining a deeper understanding of this phenomenon and finding approaches to broaden the efficiency of BCI systems to all potential users is one pivotal challenge in BCI research.

Note that the problem of deficiency is less prominent, but still significantly existing, in some BCI approaches based on event-related potentials. For results on large-scale studies and performance-related demographics, see Allison et al. (2009) for an SSVEP- and Guger et al. (2009) for a P300-based system. In SMR-based BCI systems deficiency is encountered regardless of whether a machine learning or an operant conditioning approach is used (Kübler and Müller, 2007). The actual rate of deficiency is difficult to determine, in particular since only a few BCI studies are published that have a sufficiently large number of participants who were not prescreened for being potentially good performers. There rate of deficiency in SMR-based non-invasive BCI systems can roughly be estimated to be about 15–30% and therefore poses a major obstacle for general broad BCI deployment. Still, very little is known about possible reasons of such failures in BCI control. A deeper understanding of this phenomenon requires determining factors that may serve to predict BCI performance and developing methods to quantify a predictor value from given psychological and/or physiological data. Such predictors may then help to identify strategies for future development of training methods to combat BCI deficiency and thereby provide more people with BCI communication.

With respect to SMR-based BCI systems, there is recent evidence that gamma oscillations play an important role. In a study with N = 10 participants, a relationship between gamma-power during motor imagery and classification accuracy (left hand vs. right hand) was found (Grosse-Wentrup et al., 2010). In particular, the probability of correct offline classification of a motor imagery trial was positively correlated with power in a broad gamma-frequency range (55–85 Hz) during that trial in frontal and occipital areas. Note, however, that these findings do not give rise to a predictor of BCI performance.

Such a neurophysiological predictor of BCI performance was proposed in Blankertz et al. (2010a). It is computed as band-power in physiologically specified frequency bands (below 50 Hz) from only 2 min of recording in a “relax with eyes open” condition using two Laplacian channels selectively placed over motor cortex areas. A correlation of r = 0.53 between the proposed predictor and BCI feedback performance was obtained on a large data base with N = 80 BCI-naive participants.

In a screening study, N = 80 subjects performed motor imagery first in a calibration measurement (i.e., without feedback) and then in a feedback measurement in which they could control a 1D cursor application. Basically, we observed three categories of subjects: subjects for whom (I) a classifier could be successfully trained and who performed feedback with good accuracy; (II) a classifier could be successfully trained, but feedback did not work well; (III) no classifier with acceptable accuracy could be trained. While subjects of Cat. II had obviously difficulties with the transition from offline to online operation, subjects of Cat. III did not show the expected modulation of SMRs: either no SMR idle rhythm was observed over motor areas, or this idle rhythm was not attenuated during motor imagery. For the latter case, a novel motor instruction for “quasi-movements” (i.e., movement intentions minimized to the extent that neither a mechanical limb change nor even an EMG activation remain detectable) has been proposed which led to a significant improvement of BCI performance (Nikulin et al., 2008).

Here, we present preliminary results of a pilot study (Vidaurre and Blankertz, 2010; Vidaurre et al., 2010) that investigated whether co-adaptive learning using machine learning techniques could help subjects suffering from BCI deficiency (i.e., being Cat. II or III) to achieve successful feedback. In this setup, the session immediately started with BCI feedback. In the first three runs, a subject-independent classifier pretrained on simple features (band-power in alpha-frequency (8–15 Hz) and beta-frequency (16–32 Hz) ranges in three Laplacian channels at C3, Cz, C4) was used and adapted (covariance matrix and pooled mean; Vidaurre et al., 2008) to the subject after each trial. For the subsequent three runs, a classifier was trained on a more complex band-power feature in a subject-specific narrow band composed from optimized CSP filters in six Laplacian channels. While CSP filters were static, the position of the Laplacians was updated based on a statistical criterion, and the classifier was retrained on the combined CSP plus Laplacians feature in order to provide flexibility with respect to spatial location of modulated brain activity. Finally, for the last two runs, a classifier was trained on CSP features, which have been calculated on runs 4–6. The pooled mean of the linear classifier was adapted after each trial (Vidaurre et al., 2008).

Initially, we verified the novel experimental design with six subjects of Cat. I. Here, very good feedback performance was obtained within the first run after 20–40 trials (3–6 min) of adaptation, and further increased in subsequent runs. In the present pilot study, two subjects of Cat. II and three subjects of Cat. III took part. All those five subjects did not have control in the first three runs, but they became able to gain it when the machine learning based techniques came into play in runs 4–6 (a jump from run 3 to run 4 in Cat. II, and a continuous increase in runs 4 and 5 in Cat. III, see Figure 3). This level of performance could be kept or even improved in runs 7 and 8 which used unsupervised adaptation. Summarizing, it was demonstrated that subjects suffering from BCI deficiency before could gain BCI control within one session. In particular, one subject who had no SMR idle rhythm in the beginning of the measurement could develop it with the feedback training, see Figure 3. This fundamental finding provides a perspective for the development of neurofeedback training (NFT) procedures that might help to alleviate BCI deficiency.

FIGURE 3

Figure 3. Left: Grand average of feedback performance within each run (horizontal bars and dots for each group of 20 trials) for subjects of Cat. I (N = 6), Cat. II (N = 2) and Cat. III (N = 3). An accuracy of 70% is assumed to be a threshold required for BCI applications. Note that all runs of one subject have been recorded within one session. Right: For one subject of Cat. III, spectra in channel CP3 and scalp topographies of band-power differences (signed r²-values) between the motor imagery conditions are compared between the beginning (runs 1 + 2) and the end (runs 7 + 8) of the experiment.

2.5 Guided Practice for a Fast-Decision BCI

Typically SMR-based BCIs suffer from a long latency between intention of the user and actual BCI control. Here, we introduce a “goalkeeper paradigm” that aims at improving online BCI performance by subject training under time pressure conditions (cf. Ramsey et al., 2009).

Multi-channel EEG of eight BCI-experienced subjects was acquired while they were playing three runs (100 trials each) of a BCI-controlled computer game that imitated the task of a goalkeeper during a penalty kick. During a trial, a ball was moving from the top of the screen toward one of its bottom corners. Using two different types of motor imagery (chosen from left hand, right hand, and foot) the subjects had to control the horizontal movements of a bar at the bottom of the screen in order to block the ball. Consistent with the goalkeeper metaphor, the bar could only be moved once (like a jump) into one or the other corner. The speed of the ball increased linearly from trial to trial and over the three runs. Subjects had to catch the ball within 2500 ms (at the beginning of run 1) to 1250 ms (at the end of run 3). Late arrival in a correct corner or arrival in a wrong corner were interpreted as misses.

In order to achieve a constant goalkeeping performance, the subjects were thus required to generate faster and/or stronger ERD responses in the later runs to steer the bar quickly into the correct corner. In an offline analysis, the goalkeeping performance, the reaction times (defined as the time needed to reach the correct corner) and EEG features were analyzed in relation to the block design of the experiment.

The goalkeeper paradigm effectively increased time pressure over the three runs. Performance was measured in terms of balls caught within the first 1250 ms. Seven out of eight subjects managed to respond with increased performance from run 1 to 3 (average of 33.8 balls caught in run 1 to 41.6 in run 3, see Figure 4A).

FIGURE 4

Figure 4. (A) Brain–computer interfacing reaction times are shown for all participants (asterisk) and grand average (solid line) separately for all three runs which had increasing time pressure to enforce faster BCI decision. As “BCI reaction time” we denote the latency from cue presentation until the decision of the user as conveyed by the BCI system. (B) The time-frequency plot displays the contrast (r² difference) between run 3 and run 1.

A close analysis of time-frequency EEG features between successful trials of run 1 and 3 revealed changing EEG signs of motor activation, i.e., earlier ERD or stronger ERD in the alpha-band under time pressure, cf. Figure 4B. As a side effect, the training introduced for some subjects an additional ERD in the beta-band (which had not been used for feedback). Earlier re-synchronization (ERS) could be observed for some subjects in run 3, where trials were shorter.

2.6 Exploiting Prestimulus Mental States for Better BCI Performance

Quantification of oscillatory brain activity in different frequency bands has already been widely used in the investigation of mental states. More precisely, the influence of these rhythms preceding a task on outcome performance has been extensively studied and different effects have been reported. A first body of evidence links medium and lower amplitudes in the alpha-frequency band (7–14 Hz) to better perception in somatosensory and visual discrimination tasks (Pfurtscheller and da Silva, 1999; Hanslmayr et al., 2005; Palva and Palva, 2007; van Dijk et al., 2008). As high activity in this frequency band is hypothesized to represent an idle state of cortical structures, i.e., no active processing (Pfurtscheller and da Silva, 1999), this effect can be explained by the fact that the sensory cortices involved in the task need to be in an appropriate excitation stage to process the upcoming stimulus. On the other hand, higher amplitudes over the sensorimotor cortices are correlated with better sensorimotor processing (Del et al., 2007), but less accurate inhibition of motor responses (Mazaheri et al., 2009). This suggests that a higher relaxation state of the motor system leading to higher inhibition could cause the motor system to be less responsive to signals from other regions and thus induce more straight forward processing. Concerning cognition, better performance has also been shown in the case of stronger prestimulus alpha-frequency band amplitude (Neubauer and Freudenthaler, 1995; Klimesch, 1999). Some groups even demonstrated that cognitive performance could be increased if the amplitude of the prestimulus alpha rhythm was enhanced artificially, e.g., by external stimulation (Klimesch et al., 2003; Hanslmayr et al., 2005). This is in line with the idea that more inhibition allows less signals from other brain regions to reach the neuronal network performing the task and thus induces better processing. Put together, this demonstrates a clear difference in the activation requirements of the involved brain networks between strictly perceptual tasks and more complex cognitive or motor processing tasks. In this subsection and later on, we will focus on complex cognitive and motor tasks for example motor imagery (cf. Maeder et al., 2010).

In BCI technology, SMR modulations induced by motor imagery are commonly used as task. However, performance varies extensively from trial to trial and also between sessions in the same subject. According to evidence presented here, part of this variability could be attributed to ongoing fluctuations in the SMR rhythm, see also Section 2.4.

To investigate in detail the effect of prestimulus mental states on BCI classification performance, we conducted a study analyzing the influence of prestimulus SMR amplitude on timing and strength of motor imagery induced SMR modulations.

We used data from 30 naive subjects performing left and right motor imagery in a standard cued paradigm and split feedback trials into two groups on their prestimulus band-power (high- and low-group). All trials were classified offline in sliding time intervals of 1000 ms duration as used, e.g., in cursor control, and classification accuracies were averaged within the two groups and across subjects.

We found that the classification error for the high group was lower than for the low-group over the whole trial length, see Figure 5 (left). Interestingly, this effect can be attributed to the ipsilateral rather than to the contralateral hemisphere, see the grand average ERD curves in Figure 5 (right), where the curves representing the ipsilateral side exhibit a higher difference in the post-stimulus interval than the one for the contralateral class. Figure 6 (left) displaying the ERD/ERS patterns over the whole scalp, clearly shows that the ipsilateral hemisphere stays in a higher synchronization level in the high group, that is not reach in the low-group.

FIGURE 5

Figure 5. Left: Classification error for two groups of feedback trials with high and low prestimulus SMR amplitude. Classification was performed on a 1000 ms sliding window with 50 ms overlap and significant differences are denoted by “*” (black: p < 0. 01, gray p < 0.05). Right: ERD for the different groups of the two classes (high and low for left and right).

FIGURE 6

Figure 6. Left: Scalp distribution of the ERD in the prestimulus interval (−1000 to 0 ms) and the post-stimulus interval where the significant change in performance was observed (550–2650 ms).

Although the reason for better discriminability in the high group still has to be explored in detail, we conjecture it to be due to a better relaxation state of the sensorimotor system leading to an ERD focalized to the contralateral hemisphere and an ERS (idling) over the ipsilateral one. These observations are in line with the findings by Mazaheri et al. (2009): high power in the alpha-frequency band makes the motor cortices immune to external inputs and thus are expected to generate clearer spatial patterns, which implies that they are also easier to classify.

Our findings suggest to explore NFT to increase users’ SMR amplitude and thus take advantage of the faster and more accurate classification in later BCI applications. But the effectiveness of the neurofeedback procedure, as well as the stability of its effects will have to be investigated thoroughly.

3 Mental State Monitoring

In the context of BCI technology, it is crucial to deal with the fact that brain signals exhibit an enormous trial-to-trial variability, even for constant behavioral performance. But when we turn to more difficult sensorimotor or cognitive tasks like the detection of peri-threshold stimuli, to decisions in just-notable difference discrimination, or to high-load memory tasks, also on the behavioral level we can observe considerable moment-to-moment fluctuations in reaction to the very same stimuli. Many studies in cognitive neuroscience have set out to find neuronal correlates that explain this variability (e.g., Fernández et al., 1999; Thut et al., 2006; Chen et al., 2008; Mathewson et al., 2009; Schubert et al., 2009). In particular, those studies that identify predictors from prestimulus intervals in the ongoing EEG for the performance in the subsequent task are potentially relevant for novel applications of BCI technology.

Combining such results from cognitive neuroscience with systems that allow the detection of specific mental states in real-time may eventually lead to devices that allow to optimize human performance. For example, a neurotec-enhanced vocabulary trainer may adapt the presentation of new word pairs to a moment of time in which the user is in a good mental state for memory encoding, see Guderian et al. (2009). During periods of attenuated attention, a learning game is probably more effective than drilling vocables.

On a more basic side, the way of how neuroscience and psychophysics experiments are performed may be extended in an important aspect. While upto now, participants are presented with a more or less preprogrammed sequence of stimuli (subject to some random factors), it becomes possible to adjust the presentation of stimuli to the momentary mental state of the subject.

While this area is still largely to be explored, we review here some initial approaches that show the application of BCI technology for mental state monitoring in settings that are relevant for real-world applications. See Zander and Jatzev (2009) for a general discussion of usage of BCI technology to detect covert user states. Note, that mental state monitoring has also medical use, like treating ADHD patients with neurofeedback based on an attention monitoring system, see Hamadicharef et al. (2009).

When aiming to optimize the design of user interfaces or, more general, of a work flow, the mental state of a user during task execution can provide valuable information. This information can not only be exploited for the improvement of BCI applications, but also for improving industrial production environments, the user interface of cars and for many other applications. Examples of these mental states are the levels of arousal, fatigue, emotion, workload or other variables the brain activity correlates of which are (at least partially) accessible by measurement. The improvement of suboptimal user interfaces reduces the number of critical mental states of the operators. Thus, it could lead to an increase in production yield, less errors and accidents, and avoid user frustration.

Typically, information collected about mental states of interest is exploited in an offline data analysis and leads to a redesign of task or interface. In addition, a method for mental state monitoring that can be applied online during the execution of a task might be desirable. Traditional methods for capturing mental states and user ratings are questionnaires, video surveillance of the task, or the analysis of errors made by the operator. However, questionnaires are of limited use for precisely assessing the information of interest, as the reported answers are often distorted by subjectiveness. Questionnaires cannot determine the quantities of interest in real-time (during the execution of the task) but only in retrospect; moreover, they are intrusive because they interfere with the task.

As a new approach, we propose the use of EEG signals for mental state monitoring and combine it with BCI technology for real-time data analysis and classification. With this approach, the brain signals of interest can be isolated from background activity as in BCI systems; this combination allows for the non-intrusive evaluation of mental states in real-time and on a single-trial basis such that an online system with feedback can be built.

3.1 Attention

Mental state monitoring is of particular interest in safety-critical applications where human performance is often the least controllable factor. For example, consider that fatal car accidents are one of the leading causes of death in the United States (Mokdad et al., 2004; Mokdad et al., 2005; Subramanian, 2007), and the leading cause among children (9–18 years) worldwide (Xu et al., 2010). The two main causes for crashes are distraction of (visual) attention and lapses in vigilance due to fatigue. For assessment of such mental parameters, physiological measures have been developed, blink and heart rate being the most widely used among them (e.g., Papadelis et al., 2007). Although EEG-based markers might more directly reflect cognitive processing, they have been considered less often in driving applications so far (see Brookhuis and de Waard, 1993; Otmani et al., 2005; for EEG-based investigation of attention in car driving, and the reference in Section 3.3 concerning workload).

Parieto-occipital alpha-waves are believed to be linked to idling of the visual cortex, which is the assumed default mode if no visual information is processed. Thus, high alpha-power signifies potentially dangerous brain states with reduced visual attention. In the extreme case of sleep, alpha activity is most pronounced, and so-called sleep spindles can be observed even by the naked eye. However, microsleep detection from EEG is uncalled-for, since technically more simple systems based on eyelid closure detection already achieve good results.

On the other hand it is understood that the danger of driving errors increases already a certain time before microsleep onset. During that period, responsiveness to sudden unexpected events will be degraded due to the driver’s drowsiness. A similar degradation is expected to happen during longer periods of monotonous driving. The challenge for driving assistance systems is to detect such subtle deficiency in vigilance occurring while driver’s eyes are open. Importantly, incidence of being in such an inattentive mode might be hard to infer from facial expression and eye movements alone, while neurophysiological measures could provide added value.

In a recent study the potential use of EEG-derived oscillatory features for driving assistance applications was investigated by relating band-power to driving performance in a realistic simulated scenario (Schubert et al., 2008). The experiment was designed to account for the complexity of driving, which requires attentiveness to both visual and auditory stimulation. Consequently, our investigation included a whole range of frequency bands and topographic regions-of-interest, covering alpha activity from visual cortex as well as oscillatory signals related to other cognitive systems.

Eleven right-handed male subjects, all possessing a driver’s license, participated in the study. They had to perform three different primary tasks. In the first condition (K0) the task was simply to fixate at a cross placed in front of the subject. In the second condition (K1) the cross was replaced by a video presentation of a driving scene (passive driving). The third condition (K2) finally required active driving, using a steering wheel. The subject had to perform lane changes as quickly as possible upon presentation of corresponding signs. In all three conditions, high/low tone auditory stimuli (oddball, ratio 1:5, ISI 5–6 s randomized) had to be responded to as quickly as possible as a secondary task by pressing buttons attached at left/right thumbs, respectively. The time needed for pressing the appropriate button was regarded as a substitute for driving performance. The experiment was divided into four blocks, each containing 10 min intervals under K0 and K1 conditions, as well as 20 min of K2 driving. During the whole experiment, EEG from 128 channels was acquired.

The data underwent post hoc statistical analysis using repeated measures ANOVA. For the response times of the secondary task, a model with factors stimulus (frequent vs. infrequent) and condition was considered. Furthermore, the dependence of band-power on the factors reaction time (40% fastest/slowest), condition and electrode location (subdivision into groups of 22 frontal/central/parieto-occipital electrodes) was investigated. Band-power was calculated in the theta-, alpha-, beta-, and gamma-band by applying an FFT within the 3500 ms prestimulus interval.

Reaction times were significantly larger in the driving condition than during fixation, and reactions to frequent stimuli were significantly faster compared to infrequent stimuli. Alpha- and gamma-power decreased gradually from K0 to K2 conditions. Most importantly, within each condition alpha-power was significantly higher for short, compared to long auditory reaction times. At first sight, this seems in conflict with the established role of high alpha-power as a marker of fatigue, as introduced above. However, it is in line with a study, in which a positive correlation was found between alpha-power and increasing task demand (Cooper et al., 2003). We suggest that for competing auditory and visual tasks, fast reactions to auditory stimuli can be performed only at the expense of a less efficient engagement of visual processing as revealed by a diminished alpha suppression. Our data show also that the potential range for attention-related alpha modulation gets smaller with increasing visual flow, i.e., during driving a large amount of attention is inevitably bound to the visual system.

While this study is itself not an application of BCI technology, it points a way for future applications in driving assistance systems, and the results point out why subtle methods, as developed in the field of BCI, are required: inattentiveness cannot be detected as such, but only with respect to a certain input modality which in case of driving is the visual domain. Furthermore, the neural correlates differ considerable between individuals, such that a specific calibration of the detector is needed.

3.2 Monitoring Performance Capability

Monitoring of mental states such as performance capability or task engagement can be of interest for industrial applications. A pilot study (Müller et al., 2008) with four participants evaluated the use of EEG signals in such a setting. The aim was to investigate the net effect of performance in a application oriented scenario. By choosing to simulate a real-world application, we accepted that different psychological concepts were lumped together, amongst those fatigue, concentration, task engagement. The design was not intended to disentangle those states, but rather to monitor the continuous performance capability of an operator.

The experimental paradigm simulates a security surveillance system where the sustained performance ability of the user in a monotonous task is crucial. The objective was to calibrate the BCI system to the individual user in order to recognize and predict mental states that correlate with a high or a low number of performance errors of the subject. In the following we will use the term concentration for this concept.

Participants had to rate 2000 (simulated) X-ray images of luggage objects as either dangerous or harmless by a key press with the left or right index finger after each presentation, see Figure 7. EEG was recorded from 128 channels at 1000 Hz. The session was divided into 10 runs of 200 trials each. Due to the monotonous nature of the task and the long duration of the experiment, they were expected to show a fading level of arousal, which impairs concentration and leads to more and more erroneous decisions during later blocks.

FIGURE 7

Figure 7. Stimuli used for the suitcase inspection study. The upper row shows three examples of (simulated) X-rays of suitcases that do not contain a weapon. They had to be discriminated from suitcases in which there is a weapon hidden, like the three in the lower row (machine pistol, knife, and axe).

The time course of erroneous decisions taken by a participant was smoothed in order to form a continuous measure for arousal. This measure is hereafter called the error index and reflects the subject’s inability to concentrate and fulfill the security task. To enhance the contrast of the discrimination analysis, two thresholds were introduced for the error index and set after visual inspection. Trials outside these thresholds defined two sets of trials with either a rather high or low value. The EEG data of the trials were labeled as sufficiently concentrated or insufficiently concentrated depending on these thresholds for later analysis. Figure 8 shows the error index. The subject did perform nearly error-free during the first blocks, but then showed increasing errors beginning with block four. However, as the blocks were separated by short breaks, the subject could regain concentration at the beginning of each new block at least for a small number of trials.

FIGURE 8

Figure 8. Left: Comparison of the concentration insufficiency index (CII, dotted curve) and the error index for the subject. The error index (the true performed errors smoothed over time) reflects the inverse of the arousal of the subject. Right: Correlation coefficient between the CII (returned by the classifier) and the true performance for different time shifts. Highest correlation is around a zero time shift, as expected. Note that the CII has an increased correlation with the error even before the error appears.

As neuronal correlate of decreased concentration, a left parieto-occipital increase of power in the alpha-frequency range was found. For a more detailed physiological analysis please refer to the original paper (Müller et al., 2008). Based on the contrast between periods of high and low error index, a so-called concentration insufficiency index (CII) was derived from EEG data. The output of the CII measure for each trial is plotted in Figure 8 together with the corresponding error index. It can be observed that the calculated CII mirrors the error index for most blocks. More precisely, the CII mimics the error increase inside each block, and in blocks 3 and 4 it can anticipate the increase of later blocks, i.e., out-of-sample. For those later blocks, the CII reveals that the subject could not recover full arousal during the breaks. Instead, (s)he shows a short-time arousal for the time immediately after a break, but the CII accumulates over time.

The correlation coefficient of both time series with varying temporal delay is shown in the right plot of Figure 8. The CII inferred by the classifier and the errors that the subject had actually produced correlate strongly. Furthermore, the correlation is high even for predictions that are upto 50 trials into the future.

Using a more abstract experimental paradigm, Makeig and Jung found increased theta (4–6 Hz) and decreased gamma (>35 Hz) activity to be predictive of upcoming failures in a difficult auditory detection task, see Makeig and Jung (1996). More recently, the so-called error-preceding potentials have been investigated, which are (changes in) event-related potentials that foreshadow behavioral errors, see Eichele et al. (2010).

3.3 Workload

In the previous sections we exemplified ways in which BCI technology may help to improve human performance. In contrast, the approach presented here can be used to improve the design of products. We discuss a method for real-time monitoring of mental workload, and how it can be used for neuro-usability. Beyond this aspect, there is also a long-term perspective in which the workload monitor technology could also be used to improve the human performance: real-time measures of mental workload could be incorporated in future cars in order to reduce distractions (e.g., a navigation system is switched off during periods of high workload) to a minimum when the driver’s brain is already over-loaded by other demands during potentially hazardous situations.

In the development of many new products or in the improvement of existing products, usability studies play an important role. They are performed in order to measure to what degree a product meets the intended purpose with regard to the aspects effectiveness, efficiency and user satisfaction. A further goal is to quantify the joy of use. While effectiveness can be quantified quite objectively, e.g., in terms of task completion, the other aspects are more intricate to assess. Even psychological variables consciously inaccessible to the subjects themselves might be involved. Furthermore, in usability studies it is of interest to perform an effortless continuous acquisition of usability parameters whilst not requiring any action on side of the subject as this might interfere with the task at hand. For these reasons, BCI technology could become a crucial tool for usability studies in the future.

One criterium for the usability of a car is the mental workload that is required from the car driver. If the manufacturer plans to endow the car with a new feature that uses an elaborate man–machine interface technology, the producer should demonstrate that it does not distract the driver from the traffic (i.e., mental workload should not be unduly increased when the feature is used). If a manufacturer claims that a novel device relieves the driver from workload (e.g., by means of an automatic distance control), this effect should be “proven.” In both cases, neurophysiological monitoring of mental workload could provide an objective measure.

Since there is no ground truth available on the cognitive workload to which the driver is exposed, we designed a study¹ in which additional workload was induced in a controlled manner. For details, confer Kohlmorgen et al. (2007). There are several precursory studies that derived measures of workload under laboratory conditions (e.g., Gevins et al., 1998; Berka et al., 2007; Sassaroli et al., 2008) or in the context of real operational environments (e.g., Sterman and Mann, 1995; Hankins and Wilson, 1998; Lin et al., 2005), but these have all been limited to offline analyses.

In our study, EEG was acquired from 12 male and 5 female subjects while driving on a highway at a speed of 100 km/h (primary task). Second, the subjects performed an auditory reaction task: one of two buttons mounted on the left and right index finger had to be pressed as quick as possible according to a given vocal prompt (the German words links (left) and rechts (right)) which was given every 7.5 s. This secondary task mimicked the interaction with a car’s electronic device. Furthermore, the obtained reaction times were used as a measure of the driver’s free cognitive capacity. This task was chosen to be simple and to have long inter-task periods such that it would not contribute substantially to the overall workload. Finally, in every second block of 2 min there was a tertiary task to induce workload. Two different conditions were used.

Mental calculation task (MC): Silently subtract iteratively 27 starting from a given number, that was randomly chosen between 800 and 999.

Auditory task (AT): Follow the story of an audio book while ignoring a simultaneously played news broadcast. For verification, a question related to the content of the audio book was ask after the end of the block.

These tasks had to be performed during blocks of 2 min duration (high workload condition) which were followed by blocks without tertiary task (load workload condition), see Figure 9.

FIGURE 9

Figure 9. Experimental paradigm. The tertiary task was used to induce two different types of cognitive workload. An auditory task (AT) or mental calculation (MC) had to be performed in blocks of 2 min (high workload condition) interleaved with blocks of two duration without tertiary task (low workload condition). One run consisted of three pairs of blocks of high and low workload condition.

In initial calibration phase (two runs with auditory task and two runs with calculation task), the developed BBCI workload detector was adapted to the individual driver. Roughly, the workload detector classified spatial patterns of band-power in subject-specific frequency band. Initially, unstable channels and channels presumably containing muscle or eye movement artifacts were removed. Then different parameter configurations (frequency bands, sets of channels, spatial filters, hysteresis thresholds) have been validated and the setting that provides the best discrimination of the high vs. low workload conditions was selected. For a detailed description of the algorithms, see Kohlmorgen et al. (2007).

Using this approach, the system was able to continuously predict the cognitive workload of the driver online. The accuracy of the detector with respect to the induced levels of workload was on average above 70% but varied considerably between participants from 50% (chance level) to 95.6%, see the example output in Figure 10 for the participants with the best detection results. This information was used in the application phase (also two runs with auditory and two with calculation task) to switch off the auditory reaction task when high workload was detected (“mitigation”).

FIGURE 10

Figure 10. The exact time course of the classifier output for the best performing subject (lower panel), and the corresponding binary high/low workload indication used to control the mitigation (middle panel), in comparison with the true high and low workload conditions (upper panel) for auditory workload (95.6% correct).

As a result of the mitigation strategy, the reaction time in the application phase was on average 100 ms faster than in the (un-mitigated) calibration phase (Kohlmorgen et al., 2007). The improvement in performance during the application phase can be explained by the fact that the workload detector successfully predicted periods of potentially reduced reactivity and exempted drivers from reacting during increased workload.

In this study, separate classifiers have been used for the workload induced by the AT and the MC type of tertiary task. The spatial maps of the classifier weights differed between those modes, but there was no systematic evaluation to dissociate between those types of workload.

Note that the high inter-subject variability, which is a challenge for many BCI applications, comes as an advantage here: for neuro-usability studies, top subjects (with respect to the detectability of relevant EEG components) of a study can be selected according to the appropriateness of their brain signals.

3.4 Cognition of Music

In this section, we show how BCI technology can be applied to study the cognition of music. In particular, it is demonstrated that an advanced method for single-trial ERP classification, that was developed in BCI research (Blankertz et al., 2010b) allows to reveal unconsciously perceived structures of music (Sturm et al., 2010).

Consonance between chords plays an important role in the perception of the complex concept of tonality. From a sensory point of view, degrees of consonance can be distinguished regarding psychoacoustic features, e.g., pitch distance, pitch commonality or sensory dissonance within each chord. From a cognitive point of view the consonance of chords can be judged regarding concepts from music theory like key relations or musical syntax. Inspired by Krumhansl’s probe tone experiments (Krumhansl and Kessler, 1982), the goal of this study is to find neural correlates of the processing of consonance in chord progression and the related perception of musical structure. As differences in corresponding ERP components are on a very small scale, advanced methods for ERP single-trial analysis, as developed within BCI research, are required for the extraction of the neuronal correlates.

Thirteen subjects, all musically active to a varying extent, took part in the study. Experiments consisted of several blocks of acoustic presentation of continuous sequences of major triads in root position. A chord was repeated 7–11 times before changing to a new chord of the chromatic scale in a random fashion. The subject’s task was to rate the goodness of fit of the new chord with respect to the preceding chords on a scale from 1 to 7. This resulted in a modified oddball paradigm with the new chord as deviant and the last preceding chord as standard stimulus in a continuous sequence of chords.

The subjects’ ratings were inhomogeneous but can be attributed to two main influences: (1) pitch distance and (2) structures related to models of music theory. Analysis of ERP data revealed that the N2b-P3 complex shows the differences between the classes deviant and standard most pronounced. A state-of-the-art classifier (Blankertz et al., 2010b), borrowed from BCI technology, was trained to discriminate deviant vs. standard ERPs. This classifier was then applied to deviant trials only and the output was averaged for the 11 subgroups according to the ascending interval between the root of the standard stimulus and of the deviant stimulus, resulting in an 11 dimensional profile – the neuronal correlate of a subject’s rating profile. These neuronal profiles formed for most subjects the same structure as in models of music theory. Strikingly, these structures were also found in subjects, whose behavioral data did not reveal them, suggesting an unconscious perception at this stage of processing, see Figure 11.

FIGURE 11

Figure 11. Left: Chord distances according to Lerdahl’s theory of tonal pitch space (Lerdahl, 2001). Right: The blue colored curve shows the subjective rating of participant VPcab which reflects only the distance of the fundamental tones, but does respect harmonical aspects. The orange colored line shows the output of the classifier. If reflects much better the musical structure of the stimuli than the behavioral data. (In the labels on the x-axis small and large font size corresponds to minor and major intervals).

Different complex acoustical features of chords are processed in a sensory and a cognitive way in the brain. The inhomogeneity of the subjects’ behavior, however, suggests that the ability to utilize these processes substantially differs between subjects even in a subgroup of 13 musicians. Utilizing machine learning based ERP analysis, developed in the framework of BCI research, can reveal at which stage of neuronal processing the perception of music is affected differently in certain subjects.

4 BCI For Entertainment

Finally, we discuss applications of BCI technology that have a different flavor than the ones discussed above. Here, we present some entertainment applications in which the appeal arise through the fact that they are controlled directly from the brain, like brain painting or games involving BCI control.

4.1 Media Applications

Media applications are equally attractive to use for healthy subjects and patients. It is worth to take a closer look at typical media of activities like the managing of photo-, video-, and music collections, web surfing, the sharing of media with friends and relatives, painting, presenting photos or videos in small shows to others, preparing playlists for later use, and of course the consumption of music and media for pure self-entertainment or edutainment. On a more conceptual level, these activities can be categorized in exploration, social interaction, self-expression and consumption.

For severely paralyzed patients who are dependent on assistive technology like BCI to use media applications, the media applications have to take the limitations of control signals into consideration. Ideally, the user should be enabled to express himself by creative or hedonic interaction with the media, and enjoy social activities triggered by the results of his interaction with the media, although the complexity of the user interface is limited and control signals are not fully reliable. As these restrictions are very similar to those that apply to mobile media applications on small hand-held devices (Murray-Smith, 2009; Williamson et al., 2009), the development of BCI-controlled applications can profit from the community in the field of human–computer interaction (HCI) that constantly develops new interaction models for mobile use.

As the social interaction and embedding of a patient plays a crucial role for his/her quality of life, media applications have a high potential to improve the life of a patient, although they are of non-medical nature.

Although BCI-controlled media applications currently still are in their infancy, a number of interesting applications has been reported so far that go beyond the scope of text input applications like Hex-o-Spell (Blankertz et al., 2007b; Williamson et al., 2009), Dasher (Wills and MacKay, 2006), or applications inspired by the original P300 speller grid of Farwell and Donchin (1988).

The BCI-controlled web browser interface (Nessi, Bensch et al., 2007) enables users not only to browse web pages, but to access web-based services and applications in general. It is platform-independent and open source and has been used, e.g., with a two-class motor imagery paradigm. Selectable items are highlighted by a color code. To select an item the user has to generate the type of control command with the corresponding color. Step by step, the set of selectable items is reduced until a single link is chosen. More, (Scherer et al., 2007) reports Google Earth to be controlled by BCI.

Creative activity is supported by a BCI-controlled brain painting application that exploits visual P300 signals. By concentration on selectable fields of a tool matrix the user can select simple painting tools, colors, shapes, etc., position the tools on a digital canvas and paint geometric shapes (Kübler et al., 2008). The application has been used by a number of ALS patients and a healthy artist. Although restricted in the type of tools, it enables an expression of creativity while by-passing motor pathways.

4.2 Games

There is a wide range from strictly medical to completely non-medical BCI-controlled gaming applications. The design can be such that the applications is controlled by BCI alone, or that BCI is an additional input which augments a classical control. A survey that discusses approaches and requirements of BCI-controlled games on a general level is given in Nijholt (2009) and Lécuyer et al. (2008) provides a nice overview of several BCI games and virtual environment applications.

4.2.1 Games controlled by BCI only

The medical use of BCI-controlled games is quite obvious – gaming can be an excellent motivation to spend time with a BCI system in order to achieve better control. These improvements can be expected by playing games that demand long-time concentration by the subject if the game rewards a subject for an increased SNR of the EEG signal exploited for BCI control. This can be realized, e.g., by high scores for increased control speed or by a more precise timing of the control signals. Abilities acquired during gaming will have an immediate impact on the performance of the subject in other BCI applications, like spelling, environmental control, etc.

Gaming applications of this type typically have the simple character of a neuro-feedback training. Many of them are used in a research context and only a few examples can be mentioned here. The BrainBall game (Hjelm and Browall, 2000) was introduced to learn the control of the level of relaxation expressed by the occipital alpha intensity. To improve the concentration ability on blinking areas on the computer screen, e.g., an SSVEP game (Lalor et al., 2005) can be used. Event-related potentials are exploited in the MindGame (Finke et al., 2009) which translates detected P300 components into movements of a character on a three-dimensional game board. General control via motor paradigms can be trained with, e.g., BCI-PacMac (Krepki et al., 2007), the quick generation of brisk motor imagery based BCI-commands is used as feedback to train BCI reaction time in a goalkeeper game (Ramsey et al., 2009). Apart from their novel control concept (compared to standard games), these approaches do not have an outstanding attractiveness or long-time immersive character due to their simplicity whereas their usefulness for the training of patients is obvious.

A recently published gaming application that requires simple two-class control but is more complex with respect to requirements for timing precision and its physical interactions, is a BCI-controlled pinball machine (Tangermann et al., 2009), see Figure 12. This application has proven that BCI control signals derived from motor imagery can be precise enough in timing to play a fast and reactive game in real-time. Although a trade-off between timing precision and classification precision had to be found individually even for good subjects, the game was perceived as highly immersive and motivating.

FIGURE 12

Figure 12. Overview of a BCI system for the control of a pinball machine by motor imagery of, e.g., left and right hand imagined movements.

Brain–computer interfacing games involving other mental strategies for control can be used to confirm neurophysiological findings in a rigorous manner. An example that is appealing due to its ecological validity of the control paradigm is the video game Tetris (Wikipedia, 2009). In this game pieces which are falling down slowly can be moved horizontally either to the left or to the right and can be rotated in steps of 90° by the player. In the BCI-version of the game, left and right hand motor imagery is used to move the pieces and mental rotation (Ditunno and Mann, 1990) lets the piece rotate clockwise and foot motor imagery allows to drop it. While the motor commands have been extensively used in many BCI applications, the use of the cognitive task mental rotation adds in very naturally in the Tetris game. In our pilot study, we could confirm the right parietal focus (Farah, 1989; Harris et al., 2000; Heil, 2002; Roberts and Bell, 2003; Windischberger et al., 2003; Gootjes et al., 2008; depending on task and gender it may also be left parietal, Mehta and Newcombe, 1991; see also the discussion of laterality in Milivojevic et al., 2009) of neuronal activity during mental rotation, see Figure 13. A four-class classifier (three motor imagery commands and mental rotation) trained on an offline calibration run was applied online in sliding windows every 40 ms during the gaming phase. For each movement step within the tetris game, classifier outputs were accumulated and the action corresponding to the highest value was triggered in case the probability exceeded a certain threshold. This game was not systematically evaluated in a study.

FIGURE 13

Figure 13. Brain–computer interfacing controlled tetris game. Left: A volunteer is playing a BCI-controlled version of the Tetris computer game. He uses left and right hand motor imagery to move the falling pieces horizontally, mental rotation to rotate it clockwise and foot motor imagery to let it drop. Right: The map shows the activation pattern during mental rotation in the tetris game (band-power in the beta-band 18–24 Hz with red color indicating event-related desynchronization, i.e., activation of the corresponding cortical area). The right parietal focus is in line with the literature.

However, BCI-controlled gaming applications can provide even more added value in terms of social integration, as a paralyzed player can cooperate or compete with other, possibly healthy gamers. In the simplest possible scenario, a slow strategic game like, e.g., chess that does not require fast decisions, a patient and a healthy player will meet on common ground, express and compare their cognitive abilities, learn about each other via the game. By emphasizing the interaction on a mental level, the degree of virtual disability is reduced during these interactions. Cooperational games, of course, where both players interact to reach a common goal, will provide even stronger added value in this respect.

Artificially biased gaming scenarios can provide for a fair balance: in a game like speed chess or tetris, where a player clearly profits from fast and precise-in-time control commands, healthy players can deliberately be slowed down until both gamers have approximately the same level of control. This can be accomplished, e.g., by adding delays to keyboard inputs of the able-bodied person and by adding an amount of uncertainty to each control decision. Although this might not look appealing to a healthy person on a first glance, it would be a good opportunity for healthy persons in the social environment of the patient to understand better the degree of handicap that the patient has to cope with.

Both approaches (slow strategic games and quick reactive games) open up the possibility to play remotely over the internet. Joining a gaming community has the potential to create new contacts. Interaction via BCI-controlled gaming could build up patient- or mixed communities.

Planning the creation of a new game, the restricted information transfer rate has to be taken into account. The greatest challenge is to design a user interface for the game that hides the input complexity good enough from the user to be suitable, e.g., for a slow two-class control signal, but still to provide a rich enough decision space to be appealing. Partly, the lack of control accuracy can be compensated by rich and timely feedback during every control decision initiated by the user. This is a fruitful field of cooperation for the BCI research community with designers from the HCI community.

4.2.2 Games with additional input from BCI

Although using a BCI as the only source of control input for a game can be very appealing as it is considered a “cool” and still new form of interface (cf. for example the BCI-controlled pinball machine), it is, of course, possible to use BCI-control as an additional control channel for all types of games. In the near future, when hardware costs will have decreased to a suitable level, and when robustness of the EEG recording devices is increased to meet the expectations on the gaming market, healthy users will be able to explore BCI in addition to game-pad, mouse or keyboard input devices in order to enrich the interaction space. A further boost is to be expected, when sensitive and robust dry electrode systems appear on the market that are capable of recording neural signals beyond occipital alpha or muscle artifacts only. However, it is yet unclear how smoothly the simultaneous use of traditional input devices and BCI can be coordinated, and what kind of BCI paradigms prove stable enough in such an environment. Significant further research will be required that allows to correlate activities between modalities (see Bießmann et al., 2009) and moreover that can compensate selectively non-stationarities within the different modalities (see von Bünau et al., 2009).

Given the availability of affordable recording devices, methods for mental state monitoring might have a strong impact for gaming. Mental state monitoring is a field strongly related to BCI, as it uses similar analysis methods to estimate user states in real-time. Examples are the monitoring of the level of mental workload, concentration ability, the ability to react quickly, etc. (see Section 3.2). If a game engine can make use of this additional user state information, the course of the game can be changed appropriately, the complexity level of tasks can be adapted, etc., in order to increase the level of immersion and entertainment. Applying this technique would enable to tailor a game individually to the gamer.

5 Conclusion

In the past years, BCI systems have become significantly more usable and accurate through the use of modern machine learning and signal processing technology. This has allowed to consider also applications that are beyond the classical paradigms for the disabled where BCI systems have helped to restore communication ability.

Recently, building on BCI technology more general measurement devices are developed capable of assessing and decoding more generic brain states in real-time. Successful examples of such brain state detection that have been outlined in this paper are seamless measurements of workload and performance capability. An accurate analysis of the human brain state can be employed to optimize state dependent man–machine interaction. Recent studies go beyond this by indicating that it might become possible to detect states that foreshadow errors during complex cognitive decision tasks (Eichele et al., 2010) with BCI technology.

We conclude that non-invasive BCI technology may change the way that we will in the future interact with computers. This will hold for both, healthy and disabled users.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We are indebted to the reviewers whose constructive comments on the original version of this paper help to greatly improve this paper. The studies were partly supported by the Bundesministerium für Bildung und Forschung (BMBF), Fkz 01IB001A/B, 01GQ0850, by the German Science Foundation (DFG, contract MU 987/3-1), and by the European ICT Programme Project FP7-224631 and 216886. This paper only reflects the authors’ views and funding agencies are not liable for any use that may be made of the information contained herein.

Footnote

^This study was performed in cooperation with the Daimler AG. For further information, we refer to Kohlmorgen (2007).

References

Allison, B., Lüth, T., Valbuena, D., Teymourian, A., Volosyak, I., and Gräser, A. (2009). BCI demographics: how many (and what kinds of) people can use an SSVEP BCI? IEEE Trans. Neural Syst. Rehabil. Eng. 18, 107–116.