IntEr-HRI Competition:

For this stage, pre-recorded single-trial brain activity (EEG) dataset has been provided in the form of labeled training data. To download the dataset for the training data, follow this link: Zenodo Training Data Link. To download the dataset for test data, follow this link: Zenodo Test Data Link (publicly accessible on 29 May 2023). Furthermore, detailed information about the experimental setup, procedure, and dataset description can be found in our data report paper uploaded on arXiv: (Link to the paper). 

Note:

• The above-mentioned paper provides details about data recording for our complete study wherein we also recorded EMG data and some additional experimental conditions such as delayed ball squeeze and no ball squeeze. However, for the sake of the competition, only a subset of this dataset is made public. The complete dataset will be made available after the competition.

• For the first phase of the offline stage, we have provided 8 labeled training sets from each of the 8 subjects.

• The remaining 2 sets will be made available as unlabeled test data by 29 May 2023 as per the schedule.

Problem Statement:

The challenge is to train an ML model to detect the onset of the deliberately introduced errors and perform 10-fold cross-validation (within each subject)  on the labeled training data. Furthermore, the participating teams are expected to validate the performance of their trained model on the unlabeled test data.

The provided EEG training data includes all the events recorded during the experiment inside the marker file (.vmrk). However, the test data, provided during the next phase, will not include these events to simulate a real online scenario which will be a challenge for the online stage of the competition.

Registration:

All the participating teams are supposed to register for the offline stage of the competition via the ConfTool (ConfTool Link) before the Registration Deadline for the offline competition (12 June 2023, 23:59 AoE).

Note:

• Only 1 registration per team and provide the contact details of the contact person / team leader.

• This registration is for our sake in order to better organize the participating team databases. This shall not be confused with the IJCAI’23 official registration (https://ijcai-23.org/registration/) which is mandatory if you wish to attend the conference.

• To organize the online stage of the competition, we want to know if your team would be willing to travel to Macao (if selected in the top 10). Hence, it is essential that you state your willingness in the registration form under User comments (See Sample Registration Page section). 

• Similarly, each participating team is expected to choose a team name. This, too, has to be mentioned under the User Comments section of the registration form.

Submission Guidelines:

• Submit a short paper with a maximum size of 2 pages including only 1 table and 1 figure. This report shall describe your overall approach and results in the form of a confusion matrix for the training data. In addition, the results section shall also contain the results of the test data validation (e.g.: a table containing all the sample indices per set per subject). This paper shall be named as paper_teamName.pdf (only PDF format is accepted).

• Also, we expect each team to submit a results folder named test_results_teamName. This folder shall contain a text (.txt) file for each test set (total of 16 files) with the sample indices (integer; comma-separated) of the detected error onsets. An example results folder will be provided with one example .txt file along with the test data on Zenodo as per the schedule. Please ensure that everyone follows the exact format as mentioned in the example text file. Additionally, a readme file will also be provided named test_results_readme.txt.

• We expect all teams to also provide access to their source code (preferably a GitHub repository) for evaluation purposes.

• All the documents for submission should be submitted via the ConfTool (ConfTool Link) used for registration. The submission portal only accepts one .zip file. Thus, the test_results_teamName folder as well as the paper_teamName.pdf should be copied inside a new folder named submission_teamName and compressed into a .zip file before submission.

Evaluation Metrics:

Training phase:

The training data (8 sets per subject) shall be evaluated by the participating teams using 10-fold cross-validation within each subject and the confusion matrix ([True Positive, False Negative; False Positive, True Negative]) has to be reported. 

Testing phase – Scoring:

Each of the submitted sample indices (6 per set) will be converted to the corresponding time points in milliseconds and compared against the ground truth (reference) and the temporal error (difference) in milliseconds will be calculated up to a maximum time point limit (1000 ms). If the detection happens outside the maximum time point, a penalty will be added to the team’s total (the penalty would equal the maximum time point limit i.e. 1000 ms). The error term will be summed up across all sets and all subjects and a grand total will be thus generated. This grand total will be used to determine the winners.

Note:

The detection should not happen before the actual error was introduced. Hence, a penalty(maximum time point value) will be added to the team’s total for each such instance. 

Disclaimer!!

The Early registration deadline for attending IJCAI’23 on-site is on 19 June 2023 and the Late registration deadline is on 19 July 2023. There would be a slight increase in the registration fees after the early registration (https://ijcai-23.org/registration/). 

We would not be able to finish the offline stage of the competition before the Early registration deadline but we will be providing the results before the Late registration deadline to ensure that the selected teams have enough time to decide and register for the conference. It is important for us that the participating teams are made aware of this. 

A maximum of 10 teams with the best results from the offline stage will be selected for this stage. Here, the teams will get an opportunity to test their approaches on our experimental setup in real-time.

During the competition, unlabelled single-trial EEG data will be continuously streamed from an experimental session. The participating teams are expected to have a model capable of detecting the onset of the introduced errors in the orthosis movement from the streamed EEG data.

Location and Competition Format:

The online experiment will take place off-site on the 9th of August 2023 and the data will be streamed online such that all the participating teams can access the data remotely in real-time. We strongly advise the participants of the online competition to ensure that they have a stable network connection on the day of the experimental session. The experimental protocol will be designed in such a way that small latency in the data transfer would not affect the performance or the quality of the results of the participants. All the participants are advised to attend the dry run session on the 8th of August 2023 to test the data streaming pipeline before the competition day.

AQ59D will be the subject for the experiment. The goal of the online stage is to continuously detect the introduced errors during the operation of the active orthosis device (same experimental setup and procedure as in the offline stage). However, there will be 2 scenarios where the performance of the competing teams will be evaluated –

1. detection of errors with a direct response (pressing the air-filled balloon as soon as the error is felt)
2. detection of errors with no response (the subject feels the errors  but does not press the balloon)

Therefore, the experimental conditions of scenario (1) are equivalent to the conditions of the offline competition, whereas scenario (2) does not include explicit feedback by pressing the air-filled balloon. The online classification results of both scenarios will be evaluated to determine the winning team of the competition.

Remote Online Data Streaming Protocol:

• Setting up a VPN Tunnel:
  

  In order to access the data stream, the qualified teams need to connect to our private network via a VPN tunnel. This could be done with the help of a cross-platform tool called Wireguard VPN Tunnel. This tool is available for all major OS (Linux, Windows, and macOS). Each team will be provided a unique configuration file (xyz.conf) needed to connect to our network.

  Installation and Procedure : 

  • Linux

The required packages can be installed using the following command :

$ sudo apt install wireguard-tools wireguard resolvconf

Next, navigate to the directory where you have saved the VPN config file (xyz.conf) and enter the following command to activate the VPN tunnel:

$ wg-quick up xyz.conf

If you wish to deactivate the VPN tunnel, use the following command instead:

$ wg-quick down xyz.conf

• • Windows / macOS

For Windows and macOS, there is a Wireguard app that provides an easy GUI to activate and deactivate the VPN tunnel. At the bottom of the dialog box, there is an Add Tunnel button where you can select the provided VPN config file (xyz.conf). Then all you got to do is to click on the Activate button to activate the VPN.

• Lab Streaming Layer (LSL) :
  

  We will be using LSL coupled with the VPN tunnel for streaming the data online. The stream would consist of 65 channels of data (64-channel EEG data chunks and the 65th channel would contain the markers). Also, in parallel we expect each team to send us the sample indices and time stamps as and when they detect the introduced errors as an IP request. Demo Python scripts for receiving and sending data would be provided to each team with a unique secret access code along with an LSL config file (xyz.cfg).

  For more information regarding the Lab Streaming Layer, please refer to the User Guide.

  Installation:

  There are a few packages that are required to be installed before you can use the LSL.

  • Python interface for LSL (pylsl)

In order to install pylsl, use the following command:

$ pip install pylsl

• • LSL Library (liblsl)

In order to install liblsl, visit this site. Download and install the correct software package depending on your OS or Linux distribution/version.

N.B.: The sample code for reading the LSL stream is provided on our GitHub page along with a config file (LiveAmpIjcai.cfg) which provides the 64 channel names for the EEG stream and some metadata.

Evaluation Metrics:

There are 3 different metrics that will be used to determine the final performance score for each team.

• Balanced Accuracy (BA_{cc})
• Difference between the predicted error time and the ground truth (t_{err})
• Computation time (t_{comp})

The final performance score will be a weighted average between the BA_{cc} and a consolidated time score (t_{score}) with 70% weightage given to the BA_{cc}.

Calculating the Balanced Accuracy (BA_{cc}):

The 4 terms that constitute the confusion matrix are True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). Calculating these terms is essential to determine the Balanced Accuracy of the classifier models.

BA_{cc} = \frac{\left(\frac{n_{TP}}{n_{TP} \ + \ n_{FN}}\right) + \left(\frac{n_{TN}}{n_{TN} \ + \ n_{FP}}\right)}{2}

N.B.: The boundaries of each movement trial (flexion or extension) will be the basis for determining the above quantities. Any predictions before the start of the movement trial or after the end of the movement trial will be ignored for the error in question. This means that if there are multiple false classifications within a movement trial, it would be considered a single FP. Furthermore, if there are multiple TPs and multiple FPs within a movement trial, it will be considered as a single TP (one nearest to the error will be used for calculations).  

Calculating the time score (t_{score}):

For calculating the time score, 2 quantities will be considered -> t_{err} and t_{comp}

• • t_{err}

This quantity represents the difference (in ms) between the predicted error-index and the ground truth. This is exactly the same quantity that was used in the offline stage to calculate the performance score.

• • t_{comp}

This quantity represents the time taken by the trained classifier model to classify a sample as an error. This quantity is equal to the duration between the instant when a sample was received and the moment when it was classified as an error. This quantity is added to encourage and reward real-time classification of introduced errors, which is the main purpose of this competition. Thus, t_{comp} \in [0, 1000 ms] would be the ideal case but t_{comp} \in [0, 3000 ms] will be acceptable. If  t_{comp} > 3000 ms, the classification will no longer be a TP.

The calculation of the t_{score} is a 2-step process.

• • The first step involves calculating individual time scores (t_i) for each TP. For this, logistic regression will be used. The logistic function would start from t_{err}. This means that

f(0) = t_{err}

f(3000) = 1000

Thus, for a TP, the t_{err} will be the starting score and the t_{comp} aspect will be introduced to this score via the logistical function. Thus, the X-axis for the logistical function would be t_{comp} (D :  { x | x \in [0,3000]}) and the Y-axis would be t_{i} (R :  { y | y \in [t_{err},1000]}).

• • In the second step, all the individual time scores (one each for each TP) will be summed up and normalised.

t_{score} = 1 \ –  \frac{\sum\limits_{i=1}^{n_{TP}}  \ t_i}{n_{TP} \ * \ 1000}

Calculating the final performance score (FPS):

The final performance score of each team is the weighted average between BA_{cc} and t_{score}.

FPS = W * BA_{cc}  + (1 – W) * t_{score} ,

\forall W = 0.7, and FPS \in [0,1]

N.B.: In the online case, the team with the highest FPS will be the winner.

Keynotes:

As the competition has been shifted off-site, with the consultation from the keynote speakers, it has been decided to pre-record the keynotes and make them available around mid-July 2023 (See Keynotes section for embedded recorded keynotes). Hopefully, this would give ample time to the participating teams to listen to the keynotes and have an informed discussion with the speakers during the Q&A session that will be organized during the IJCAI’23 conference.

Our booth at IJCAI’23 Conference:

During the conference in Macao, a booth will be set up for the duration of 1.5 hours, during which selected top-performing teams shall present their approach and results. Additionally, an open Q&A session will be organized with the keynote speakers, providing an opportunity to exchange ideas, address concerns, clarify doubts, and engage in scientific discussions with experts in the fields of AI, Robotics, Neuroscience, and HRI.