Are robots perceived as good decision makers? A study investigating trust and preference of robotic and human linesman-referees in football

3.1 “Robot linesman in football game” as the study case

To investigate the relationships among “Trust,” “Preference,” and “Physical Appearance,” the authors employed the specific case of robot linesmen (a type of football referee) making “offside” (a type of foul play [14]) decisions in football game. This specific case is selected because of the following two reasons:

First, football game provides the perfect test ground where people’s trusts in referees are of vital importance to the game itself (if the game is fair) and the experiences of the game participants (players, game organizers, etc.) and audiences [15]. Decisions made by the referees are at the center of game fairness which always give rise to strong passions and emotions among game participants and audiences. This can potentially make it easier for us to observe the relationships between “Trust” and “Preferences,” given the different physical appearances of decision-making robots.

Second, in football, there is a push for technology to make more accurate and fairer decisions because of the prevalence of human errors from human referees [16]. Robotic officials are expected to aid in helping improve the accuracy of crucial calls during the game, and topflight football matches could be officiated by robot referees and linesmen by 2030 [17]. Thus, there are strong necessities of studying robot football referees as decision makers.

3.2 Hypothesis proposed for this study case

This study will attempt to test the following three hypotheses:

Figure 1

The four linesmen from Malle et al. that we transport to our video clips (left to right): AI, mechanical, humanoid, and human.

3.3 Predesign of online study

This online study uses the setting of football to study decision-making of a robot. The study is designed to determine whether robots are perceived as good decision makers based on which visual features and communication methods used by a robot have a higher level of average trust among participants. Participants are also asked to rate which linesman they would most prefer to make offside calls in football. The objective of this second part is to determine whether there is a direct correlation between “Trust” and “Preference.”

Before conducting the online survey, a pilot experiment was conducted on the online survey (N = 4) to improve its design. Half of the pilot participants had good knowledge of football rules and the other half had poor knowledge. In the pilot, our research team found that participants had difficulty following 8 of the 16 scenario clips at full speed and making the judgment of offsides. Consequently, the clips were slowed down to 70% running speed to make it easier for participants to distinguish for themselves whether a scenario is offside or not. Participants in the pilot study also suggested controlling for other variables concerning the linesman such as their size, position, and speed of reaction on the screen.

3.4 Participants of online study

The study was setup as an external online survey on Amazon Mechanical Turk to recruit participants. The participants were presented a link to Qualtrics where the survey was originally created. A total of 118 people participated in the study, but 14 of these were omitted for providing incomplete data or giving irresponsible answers (choosing the same answer for more than 20 questions consecutively), resulting in a sample size of 104. Of the 104 participants, 31 were female and 73 were male. The age demographics of the participants were reported as follows: 43 participants were between 20 and 29 years, 42 participants were 30–39, 13 participants were 40–49, three participants were 50–59, and three participants were 60–69 years.

3.5 Design of online study

The online survey begins by presenting a hypothetical situation to the participant: a start-up company, Ref-Tech, is trying to determine what kind of robot to use for football linesman. The study then proceeds to show the participants a 42-s video clip about how offsides work. To make sure participants understand the offside rule, participants are shown two clips (one offside scenario and another onside/nonoffside scenario) and are asked to judge whether the clip presents an offside or not.

The core of the study requires participants to view clips of offside calls (similar to the two-test scenarios) being made by linesmen of different appearances and then judge how much they trust the calls. The 11- to 17 -s-long clips, obtained from a library of offside clips used for testing purposes by the Professional Referee Organization [18], show calls that are made by four different linesmen. These four agents are taken from a similar study by Malle et al. [2] who investigated the impact of the action and appearance of a robot on people’s judgments and HR asymmetry. For this study, illustrations of the four agents – AI, mechanical robot, humanoid robot, and human – were directly extracted from the paper by Malle et al. [2] (see Figure 1) and placed into our football setting.

Research has shown that facial features, gaze, height, gender, voice, trajectory design, and even proximity to human partners all play a role in how humans respond to robots [19,20,21,22]. However, no comprehensive theory predicts which aspects of appearance matter when it comes to people trusting robot’s actions. Thus, accumulating systematic empirical research is key to understanding this relationship.

In the video, the image of one of the four linesman-referees is positioned on the side of the pitch (top left of the video that the participant watches) and makes offside calls. The participant watches a total of 16 clips of offside calls, with 4 calls being made by each of the four linesmen. Of these four offside calls made by each linesman, there is one correct offside call, one wrong offside call, one correct nonoffside call, and one wrong nonoffside call. These four decision outcomes represent, respectively, a hit, false alarm, correct reject, and miss. Offside calls are made by the linesman displaying or lifting a red and yellow checkered flag. The offside/onside call made by the linesman agent is conveyed by an “offside” or “onside” message in the video. The image stays the same when an onside call is made. The order of the type of call and the type of linesman are randomized to avoid priming effect among participants. Each participant sees the same random order from scenarios 1 to 16. Two separate independent variables in this 4 × 4 study are “type of call” and “type of linesman,” and the dependent variable is “Trust” in the linesman’s call. Table 1 shows the combination of independent variables for 16 scenarios.

The video clips, 11–17 s long, of 16 scenarios were obtained from a library of offside clips used for testing purposes by the Professional Referee Organization [18]. For each of the 16 clips of the offside calls, the participant is asked to rate how much they trust the call (see Figure 2). They do so with an empirically based scale developed to measure “Trust” in automated systems and human–machine systems based on the performed cluster analysis [23]. At the end of watching the 16 clips, the participant is asked to rank the four linesmen in order, from the linesman they most trust to the linesman they least trust.

Figure 2

Snapshot of scenario video clip with Likert scale.

The participant is then asked to rank the linesmen on how much they would prefer to have them on the football, making offside calls. At the end of the survey, open-ended questions probe the reasons behind participants’ choice for their most and least preferred linesman. As part of the final background section, which records the participants’ age and gender, participants are also asked to rate how much knowledge of football they have on a Likert scale of 0–10.

The variables “Age,” “Gender,” and “Knowledge” are not the focus of this study. The measurement of “Trust” in this study relies on the scale described in “Section F” below, and the measurement of “Preference” is based on participants’ self-reported “Preference” rankings.

3.6 Measurement used for the online study

The empirically developed scale measuring “Trust” between humans and automated systems [23] including robots was used to measure “Trust” (the dependent variable) in this study. This 7-point Likert scale possesses 12 items:

1. The system is deceptive.

2. The system behaves in an underhanded manner.

3. I am suspicious of the system’s intent, action, or inputs.

4. I am wary of the system.

5. The system’s action will have a harmful or injurious outcome.

6. I am confident of the system.

7. The system provides security.

8. The system has integrity.

9. The system is dependable.

10. The system is reliable.

11. I can trust the system.

12. I am familiar with the system.

Items 1–5 are negative questions whose answers were reversed for statistical analysis.

4.1 Correlations between “Physical Appearances” and “Trust” (Hypothesis 1)

The first variable we measured is “Trust,” and the overall “Trust” score is calculated as the mean score of the 12-item Likert scale for all the four scenarios. The Cronbach α of this scale was found to be very high: for AI linesman, α = 0.92; for mechanical linesman, α = 0.87; for humanoid linesman, α = 0.90; and for human linesman, α = 0.89. This outcome is not surprising since we used a validated scale.

Figure 3 shows no obvious trust difference for the four types of linesmen: the medians are all between 4.3 and 4.5 and the data spreads are similar. Thus, the authors cannot give a “Trust” ranking with enough statistical significance. The p value from Friedman test is 0.065 (>0.05) which indicates that no statistically significant difference in people’s trust level of these four types of linesmen.

Figure 3

Mean Likert scale scores of human–robot (HR) trust items for four types of linesmen.

4.2 Correlations between “Physical Appearances” and “Preference”

The second dependent variable measured in this study is “Preference.” Figure 4 shows the descriptive statistics of people’s preferences of four types of linesman. Participants ranked their preferences from 1 (most preferred) to 4 (least preferred) in the online study. To make it more intuitive and comparative with Figure 3, the authors reversed and linearly rescaled the data to the range of 1 (least preferred) to 7 (most preferred). In Figure 4, the medians of AI and human linesmen’s user preference level (which is 5) are much higher than the medians of mechanical and humanoid linesmen’s user preference level (which is 3). A nonparametric Friedman test of preference difference among four types of linesmen was conducted and rendered a Chi-square value of 17.70 which was significant (p < 0.001).

Figure 4

Preference levels of four types of linesmen.

The authors then performed “Nemenyi Multiple Comparison” which is a standard post hoc for Friedman test. The results are presented in Table 2. Based on test results, human linesman is preferred to mechanical linesman (p = 0.0027) and humanoid linesman (p = 0.0013) with enough statistical significance while AI linesman may or may not be preferred to any other linesmen.

By comparing Figures 3 and 4, we can easily see that although the trust levels for different linesmen are quite similar, the user preferences for the four types of linesmen are significantly different. This implies that other factors besides “Trust” probably influence people’s preferences of different types of linesmen. This deduction is supported by qualitative analysis and further discussed in section 4.4.

4.3 Correlations between “Trust” and “Preference” (Hypothesis 2) and correlation strength variations among different types of linesmen (Hypothesis 3)

Overall, the authors found correlations between “Trust” and “Preference” through Spearman’s correlation test (r (102) = −0.33, p < 0.001). To further probe the relationship between “Trust” and “Preference,” the authors also performed Spearman’s correlation test for each of the linesman conditions (AI, mechanical, humanoid, and human). Table 3 shows the correlation test results with p values and correlation coefficients of four types of linesmen.

As shown in Table 3, no correlations were observed between people’s “Trust” and “Preference” for AI linesman and Mechanical linesman. However, a low degree of correlation was observed between “Trust” and “Preference” for Humanoid Linesman (r (102) = 0.20, p = 0.039) and a moderate degree of correlation between “Trust” and “Preference” for Human linesman (r (102) = 0.30, p = 0.002). This indicates that “Robots’ Physical Appearance” is possibly affecting the correlations between people’s trust in and preference for different types of linesmen. More specifically, the more the linesman looks like a human in its appearance (no matter if it’s a real human or not), the more people relate their preference for this linesman to their trust in this linesman.

Since this experiment is not designed to draw any conclusions about causal relationships among different variables, the authors cannot run a moderated regression analysis to probe the possible interaction effects where “Robots’ Physical Appearance” is the moderator and “Trust” is the independent variable. However, the authors still think the Spearman’s correlation test results of each type of linesman in Table 3 are interesting and may inspire other HRI researchers for further investigations. Maybe in the future, a controlled in-lab experiment probing the casual relationships between “Trust” and “Preference” can offer the ideal experimental setting to investigate the potential interaction effects.

4.4 Reasons for people’s preference differences among different linesmen

In the open-ended questions asked at the end of the online survey, participants specified (in words) their reasons for their preference differences among four types of linesmen. In this section, the authors focus on the narrative reasons for the two types of linesmen with the highest median preference levels: Human Linesman and AI Linesman.

Fifty-three participants (out of 104 participants) chose human linesman as their favorite, and 39 of them reported reasons closely related to their trust in human linesman and his/her ability to make the best judgment, such as “The human got it right more often, by my estimation,” “It is a person that I would trust it (the role of a linesman) most,” “The human could see between the players and make better decisions,” “The human can understand and figure out details (and thus make better decisions),” and so on. Twenty participants chose human linesman as their least favorite, and 10 of them clearly mentioned “distrust” in human honesty (e.g., “A human can be bribed,” “A human may be betting on the game and trying to rig it.” etc.,) or human performance (e.g., “Too much human errors,” “People make mistakes,” etc.). These results support the correlation analysis in section 4.2 where “Trust” and “Preference” for human linesman are moderately correlated. Nevertheless, other reasons mentioned for preferring human linesman include “I am used to humans as a linesman,” “I prefer the traditional approach and human error is a part of sports,” and “I don’t want robots to replace humans in everything.”

Of the 104 participants, 29 chose AI as the most preferred linesman and 11 of these participants reported reasons closely related to their trust in AI linesman, including “It is a machine and will not make mistakes,” “I believe Artificial Intelligence can exceed human’s precision,” “AI, cold and logical, just sees data,” and “Seemed more honest than the rest.” Other reasons reported by the participants include “better stability” (“It seemed to be the most stable one.”) and “minimal design” (“AI is good without a needless human shape/conformation.”). In addition, 14 of 27 people who disliked the AI linesman the most mentioned its incompetency or inaccuracy in decision-making. Other reasons for participants’ dissatisfaction with the AI linesman include its poor design (“I need something more than a box to be satisfied.”), emotionlessness (“I find the AI unit to be cold and sterile.”), and bulky, physical volume (“I felt it was too big and not able to move like a human.”). These results support the correlation analysis in section 4.2 where “Trust” and “Preference” for AI linesman is not correlated. However, this does not mean that the trust issue for AI linesman is not important. Other factors (specified above) together with the trust issue should be considered for the design of AI linesman.

5.1 Possible explanations of results in Table 3

Table 3 shows a very interesting phenomenon which can be subject to multiple reasonable explanations. One potential explanation could be, the authors believe, that the more a linesman resembles human in its appearance, the more people perceive it as an agent similar to human, and the more people judge it as a moral being with positive or negative emotions (either consciously or subconsciously) [9,24,25]. For example, people will only feel pity if a machine doesn’t function very well (which may lead to distrust to the machine) but can be angry and offended if they suspect that a highly intelligent humanoid linesman purposefully cheats in the game [5]. The latter scenario can greatly influence people’s preference for this linesman. Out of curiosity, the authors performed a Friedman test comparing the differences in perceived deceptiveness among four types of linesmen using the responses from the first three questions in the “Trust” scale. The result was not significant (p = 0.20). One possibility is that our sample size is too small to detect people’s perception of deception. And again, this is only one possible theory that the authors think is interesting to consider for HRI research in general. Further research exploring robots’ appearances and the corresponding user perception of robots’ deceptiveness in various HRI contexts can be interesting. The design applications of this research will be further discussed in the section below.

5.2 Design implications

In this section, the authors will discuss the design implications based on our study results for both robot football referees and decision-making robots in general. The quoted texts are directly copied from study transcripts.

For football games, the authors would recommend having both human and AI linesmen in the game since human linesman is most preferred by participants and AI linesman complement human linesman very well based on our qualitative results. Human linesman could “see between the players,” “better understand the contexts,” “in foot details,” and “add passion to the game” while AI linesman “is cold and logical,” cannot “be bribed,” and is “more stable” than human. Moreover, human linesman has long been considered as part of the game while AI linesman can play an assistance role of “reducing human errors.” For instance, in the “2030 football game” [17], the authors would recommend having both human and AI linesmen on the field. Nevertheless, the authors believe that the final decision should be made by human linesman since human understands the unpredictable contexts and situations in a football game much better than AI.

Similarly, for decision-making tasks in general, the authors would recommend having both human and decision-making robots together in the task so that they could complement each other in the task: the human decision makers would understand more about the contexts of the tasks while the more objective AI will help to avoid human errors by being used as a support tool. Nevertheless, the authors believe that the final decisions should be made by the human decision makers since situations in real life are always important and hard to predict.

For the design of AI linesman, the authors would suggest making them not similar to humans at all – not even anthropomorphic, in any way – since (1) people do not prefer humanoid linesman, (2) the more it looks like a human, the more people focus on trust issues for their linesman preference, and (3) people appreciate the honest form factor of AI linesman, which indicates it is a machine, not human. Moreover, the authors would recommend making the AI linesman concise, elegant, light, and visibly present on the field. These characteristics were all mentioned by the participants and, as we know, may contribute significantly to people’s preferences for linesman beside the “Trust” factor. Nevertheless, making the AI linesman trustworthy, both in its form factor and in its competency, is still a factor that should be carefully considered in the design process.

Similarly, for the design of decision-making robots in general, the authors would suggest to make the physical appearances not anthropomorphic for the following reasons: (1) people may not prefer humanoid as a decision maker since people appreciate the honest physical form of the decision-making robots and (2) the more the decision-making robots look like human, the more likely people will focus on trust issues for their robot preference (e.g., people may perceive deceptiveness from humanoid robots [25]). In addition, the authors would recommend making the physical embodiment of the decision-making robots more concise, elegant, lightweighted, and even portable since these characteristics may significantly contribute to user preference for the decision-making robots. Portable decision-making robots could also facilitate humans to make onsite decisions more accurately and conveniently. Nevertheless, improving users’ trust in decision-making robots both in their physical form and in their task competency is still an important goal in the robot design.