Applied Ergonomics 100 (2022) 103670

Contents lists available at ScienceDirect

Applied Ergonomics
journal homepage: www.elsevier.com/locate/apergo

Impact of alarm management and automation on abnormal operations: A

human-in-the-loop simulation study
Richard J. Simonson a, *, Joseph R. Keebler a, Elizabeth L. Blickensderfer a, Ron Besuijen b
a
Department of Human Factors and Behavioral Neurobiology, Embry-Riddle Aeronautical University, Daytona Beach, FL, USA
b
Center for Operator Performance, Dayton, OH, USA

A R T I C L E I N F O A B S T R A C T

Keywords: Process control consists of complex states of performance that require a significant amount of operator attention
Process control and skill to manage. Automation and smart alarms can enhance these processes. This study investigated the
Workload effects of alarm management and automation on process control operator workload (NASA-TLX) and perfor­
Flare
mance via material lost through flaring outcomes. Eleven console qualified operators participated in a human-in-
Alarm rationalization
Automation
the-loop, high-fidelity simulation-based training exercise. Three levels of alarm management (no rationalization,
rationalization, and state-based smart alarm) and two levels of automation (no automation and automation)
conditions were implemented using six scenarios. A repeated measures two-way MANOVA indicated that both
alarm management schema and automation significantly affected operator workload and performance. These
results indicate that state-based smart alarm management and automation schemas may assist operators in
reducing workload and material lost through flare release during abnormal operating conditions.

1. Introduction industry. This paper describes a study that examined the impact of alarm
schema and levels of automation on operator performance and workload
Manufacturing and other complex product refinement and develop­ in a high-fidelity simulation of a chemical manufacturing plant.
ment domains (e.g., mining, oil refining, chemical manufacturing,
pharmaceutical manufacturing) actively utilize process control to
monitor processes underway and react to any abnormal or out-of-control 1.1. Alarm strategy
processes for safety and efficiency purposes (NIST/SEMATECH
e-Handbook of Statistical Methods, 2013). The complexity of these In terms of alarm strategy, rationalized alarm strategies are those
systems in the recent century has far surpassed the operator’s cognitive that review and validate the state of the processes and determine if an
abilities to perceive and attend to all pertinent pieces of information alarm meets the criteria of identifying or warning of an abnormal or out-
presented to them (Nochur et al., 2001). During states of abnormality or of-control process (Beebe et al., 2013; Rockwell Automation, 2017a).
out-of-control processes, process controls have been shown to activate Alarm practices such as rationalization have been shown to be a sig­
and elicit up to 275 alarms during the event (Hollender and Beuthel, nificant source of increased efficiency and time reductions in finding the
2007), and in one case over 300 alarms in just 5 min (Strobhar, 2012). true source of alarm activations. Specifically, it has been shown that up
This excess of information presents a multitude of challenges, mainly to 80% of alarms during an event are the result of nuisance or bad actor
due to the limited cognitive abilities of attention (Miller, 1956; Dadashi alarms, which are defined as alarms that activate as an indirect result of
et al., 2017). The discrepancy between the amount of information pre­ the source (Rockwell Automation, 2017b). These nuisance and bad actor
sented and the amount of information to which individuals can effec­ alarms can divert operator attention and/or interfere with the operator’s
tively attend leads to increased workloads, human error, and decreases acknowledgment of the source alarm within a reasonable and necessary
in efficiency (Stanton et al., 2009). Alarm schemas (e.g., rationalized time frame. While the positive effects of properly designed and imple­
and “smart alarming”) (Hollifield, 2010) and automation technologies mented alarm management schemas are apparent, the development of a
have been used to mitigate these challenges in the process control system requires careful consideration of the system, possible alarms,
past experiences, and the processes being controlled.

* Corresponding author. 1 Aerospace Blvd, Daytona Beach, FL, 32114, USA.

E-mail address: [email protected] (R.J. Simonson).

https://doi.org/10.1016/j.apergo.2021.103670
Received 20 May 2021; Received in revised form 18 November 2021; Accepted 13 December 2021
Available online 21 December 2021
0003-6870/© 2021 Elsevier Ltd. All rights reserved.
R.J. Simonson et al. Applied Ergonomics 100 (2022) 103670

A multitude of recommended alarm management policies and automation within the process control industry (as well as many other
development frameworks exist to assist control process personnel industries), and several recommendations exist for best utilizing auto­
develop such a schema (e.g., Rockwell Automation, 2017a; Brown, mation. First, current process control tasks are designed around human
2003; Veland et al., 2001), the focus of the current project is on smart cognitive abilities, processes, and their advantages and limitations.
alarming. Smart alarming is a type of alarm management that relies on a Hence, a key step when implementing automation into the process
state-based philosophy, wherein only alarms that would be applicable to control tasks is to evaluate the task structures and change those task
the state of the process would be activated (Siemens, 2018). Typically, structures as needed to suit both the automation technology, the human
the research on alarm management focuses on the reduction of the operators, and the interactions between the two (Lee and Seppelt, 2009).
number of alarms rather than on its impact on human operator perfor­ To accomplish this, recommended strategies include the use of Fitts’ list
mance. It is assumed that fewer alarms lead to better performance. One (Fitts, 1951) to consider human information processing strengths and
likely reason for the lack of performance-based research is that to best weaknesses when assigning tasks to automation/machines versus
assess the impact of alarm design on operators’ performance, a human operators, ensure the operator(s) understands the distribution of
high-fidelity simulation of the particular manufacturing process is responsibilities (i.e., what tasks the automation vs. the operator will
required. Hence, one aim of the current study was to fill this research perform), and employ operator training that teaches the operators how
gap through studying smart alarming in a high-fidelity chemical pro­ to interact with the automated tasks (Lee and Seppelt, 2006).
cessing simulation. The second recommendation is the need to account for the human
operators’ characteristics and match the automation to those charac­
1.2. Automation teristics (Lee and Seppelt, 2006). That is, instead of applying automation
to an omnibus task, it is recommended to analyze or break down the task
Automation techniques in process control have been designed to in terms of interactions with the stages of human information processing
reduce the workload that an operator must undertake when responding and, in turn, design the automation to best accommodate human
to abnormal events. An automated system is defined as “a system that cognitive performance. For example, Lee et al. (1999) provide evidence
accomplishes (partially or fully) a function that was previously, or that notification-based automation alerts as a way to capture the oper­
conceivably could be, carried out (partially or fully) by a human oper­ ator’s attention were more effective than command-based alerts at
ator” (Parasuraman et al., 2000, p. 287). Automation operates between a increasing the performance and safety of human drivers. These infor­
low and high level of intervention in tasks. Numerous taxonomies that mation interpretation messages are shown to be more effective at
categorize and classify differences in automation levels exist (e.g., providing operators a greater mental model of the situation they are
Endsley, 1987; Endsley, 1999; Riley, 1989; Sheridan and Verplank, engaged in as compared to action selection commands (Lee and Seppelt,
1978). The taxonomies emphasize that automation should match the 2006).
stages of human information processing (Popken and Krems, 2017). A third recommendation for utilizing automation is to ensure auto­
While the benefits of automation are vast and include decreased mated systems provide appropriate feedback to the operators. Adequate
workload, increased situational awareness, and improved performance feedback is essential for operators to develop accurate mental models of
(Lorenz et al., 2001), the dangers of inappropriate usage/applications of the situation and to prevent operators from out-of-the-loop performance
automation can be devastating (Reddy and Mehta, 2015). A primary (Lee and Seppelt, 2006). Display strategies that include visual forms of
issue with implementing automation into a human-machine system is data relationships, use of visual features that represents said data, and
that the automation can negatively impact the operators’ understanding providing data and its visualized forms in meaningful contexts are
of the system overall, and in turn, have a negative impact on specific task proven methods of providing intuitive feedback to human process con­
processes. Negative impacts of automation include requiring excessive trol operators (Guerlain et al., 2002).
operator monitoring of automated tasks (and, in turn, operator Finally, automation should match the mental model of the operator.
boredom), operators over-relying on the automated system, operators Lee and Seppelt (2006) provide the example that cruise control and
having low situational awareness, operators performing out-of-the-loop, assisted breaking technologies match the driver’s mental model of
and/or reduced operator trust of the automation (Popken and Krems, driving as they perform similar reactions and feedback that the driver
2017). Operators who are performing out-of-the-loop are those who are would expect. They exemplify that an automated braking system slows
unaware of the systems current process state. This operating state is the car down when an obstacle is sensed in front of the car, indicating to
typically associated with automation, as it provides a means to allow an the driver that they can enter to control-loop and brake for a more im­
operator to pay less attention to various details regarding the process mediate and controllable reaction if needed (See Table 1 for a summary
(Endsley and Kiris, 1995). Research demonstrates that unsuitably high of recommendations).
or low automation usage can cause errors and violations, which in turn, An example of the implementation of automation within process
lead to adverse events (de Visser and Parasuraman, 2011). In fact, these control industries that follow these suggestions is the usage of safe-park.
are the most frequent challenges that occur, either with an improperly Safe-park consists of a series of processes that drive the system to a
designed performance-feedback loop that generates low operator trust suitable point wherein the out-of-control process can be isolated. In this
in automation (i.e., negates the need for it) or overly high operator trust investigation, we operationalize safe-park as shutting down and
in automation (i.e., over-reliance on the automated system and infor­
mation). Additionally, another area of research, adaptive automation,
focuses on actively conforming an automated process to the operators’ Table 1
physical and mental demands (Hancock et al., 2013). While automation, Recommended methods of implementing automation.
in general, has been implemented into the process control industry, Recommendation Example
adaptive automation has yet to be employed. However, recent research Use Fitts’ list. Information processing at the information
into adaptive automation within the process control industry provides acquisition, analysis, action selection and
evidence that, during complex and information-rich scenarios, operators implementation stages.
Match automation and human Use notification-based alerts rather than
performed better and preferred high levels of automation that required
characteristics. command based-alerts.
them to progress the scenario by actively participating in the automation Provide appropriate feedback. Utilize dashboards to provide operators with up-
(e.g., forced-choice automation) (Sauer et al., 2011). These results to-date and accurate information regarding
extend prior beliefs that providing an avenue to invite the operator to process status and automation decisions.
work in-the-loop with automation may yield better results. Match automation and operator Utilize automated vehicle braking systems that
mental models. invite the operator back into the loop.
On-going research is aimed at how to most effectively implement

2
R.J. Simonson et al. Applied Ergonomics 100 (2022) 103670

isolating the out-of-control process completely. The alarm management techniques on operator performance and workload in ethylene
and automation schemas utilized in the investigation were manipulated manufacturing. Based on prior literature, the predictions were: 1) the
to determine if any significant differences were present in operator use of rationalized and smart alarm schemas would reduce operator
performance and workload outcomes. perceived workload and the amount of flaring as compared to a non-
rationalized alarm design, and 2) the use of automation (i.e., a safe-
1.3. Operator performance in ethylene production park application) would reduce both operator perceived workload and
the amount of flaring as compared to without the use of the automation.
Flaring is considered to be a natural, although undesirable, process
within the gas production and processing procedure. Upon 2. Method
manufacturing startup, shutdown, and during the manufacturing stages,
it is known that some of the product will be released through collection 2.1. Apparatus
headers where it is later burned in flare stacks (Ohio, 2014). In most
cases, the resulting flaring is due to production testing and gas man­ The quality improvement study focused on the indoor, control panel
agement, however, in some cases, flaring is a part of an emergency, operating tasks for the finishing side of ethylene manufacturing. A high-
where some abnormal or adverse event causes or necessitates the release fidelity, dynamic simulator modeled by Corys (2015) was used (see
of the product to the flare. Fig. 1). The model was for the finishing side of the process.
While flaring is often a natural and controlled process during normal Each set of monitors represented in Fig. 1 are set to display specific
operations, these emergency scenarios can release flare at possibly un­ information during a simulation-based training scenario in ethylene
controlled rates depending on the severity of the emergency for a production. The Upper two monitors visualize fixed emergency alarms
duration of time until the offending issue is addressed. During these that are always on display. The first monitor on the left in the middle
events, the processing plant is exposed to increased hazards and lose displays a video of the flare stack while the second to the left middle
their product. Additionally, potential greenhouse gasses in the form of monitor displays the equipment being used in the process. The third
carbon dioxide (CO2) and water vapor (H2O) are released into the sur­ from the left middle monitor displays alarm summary and trends while
rounding ecosystem (Emam, 2015). the fourth and fifth from the left middle displays are used to monitor
process trends during steady-state operations. All four monitors on the
1.4. Purpose of the study bottom are used to display an overview of the process and allow for the
operator to display visualizations of the process in question. During the
Alarm management schemas and automation are widely used within conditions experienced as a part of this investigation, all monitors were
the process control industries to prevent, manage, and mitigate out-of- on and displayed the information as if it were a real scenario. However,
control and abnormal processes. Published literature, industry guides, to successfully complete the scenarios participants were only required to
and government ISO’s provide guidance on how to effectively imple­ use the bottom left monitor to view a graphic of the process.
ment automation and alarming solutions, however, each industry and
process may require novel and/or unique modifications for a successful 2.2. Operators
application. Hence, a portion of process control research has been
dedicated to the investigation of these effects and the implication of the Eleven male finishing side operators currently employed at an
results for the applicable industries. The purpose of the current study ethylene refinery in Canada participated. All operators were shift
was to conduct a high-fidelity, simulation-based, human-in-the-loop workers on a five-week rotating schedule and were scheduled for their
study to quantify the effects of alarm design and automation required simulation-based training. As all operator participants were

Fig. 1. Operator monitoring a simulated control room display.

3
R.J. Simonson et al. Applied Ergonomics 100 (2022) 103670

active panel operators with an average experience of 13.18 (SD = 6.38) during a real instance of each failure. For alarm rationalization, the SME
years, they were familiar with the equipment, processes, and procedures would evaluate any alarm that would appear during the simulation and
specific to the plant and specific to their job. Prior to the start of the removed any alarm that was not justified based on their expert knowl­
scenario operators indicated their levels of alertness on a Likert-type edge of the facilities operation and the processes being simulated. For
scale from one to seven, one being exhausted and seven being more state-based alarms, the SME removed any alarm that was not related to
alert than usual. On average, the operators reported feeling relatively the operating state of the facility and then repeated the rationalization
average energy levels (M = 3.7, SD = 0.64). Additionally, the average process from the rationalization condition.
participant’s previous active shift (i.e., previous day of work) was 175 The study compared two levels of automation: presence or absence of
days (SD = 142.2) prior to his training session. the safe-park application. With the safe-park, operators would perform
approximately 8 moves in responding to the simulated events. Without
the safe-park app, operators would perform approximately 120 moves in
2.3. Simulated scenario
response to the simulated events.
Combining the two independent variables resulted in a 3 × 2 fully
The study focused on the indoor control panel operating tasks for the
within (repeated measures) quasi-experimental study with six study
finishing side of ethylene manufacturing. The simulation began with a
conditions. Thus, operators experienced three separate, simulated sce­
malfunction of the propylene refrigeration compressor (i.e., “K601
narios which included the three alarm designs (no-rationalization,
trip”), and hence the plant was in “upset” mode and not steady-state.
rationalized, and smart-alarming) in conjunction with using the safe-
When the refrigeration compressor tripped, the control room operator
park application. Additionally, the operators experienced separate,
worked to isolate the plant, and this included closing valves, shutting off
simulated scenarios with the three alarm designs (no-rationalization,
pumps, and flaring (i.e., burning off excess gases). As the operator was
rationalized, and smart-alarming) when not using the safe-park app.
isolating the plant, a valve malfunction occurred. The operators’ task
was to detect the valve malfunction while continuing to resolve the
2.5. Measures
K601 trip (see Table 2).

The measures included a demographic data form, the NASA-TLX, and

2.3.1. Valve malfunctions
amount of material released via the flare.
To avoid learning effects, six separate scenarios (one scenario per
condition) were used in the study, and each lasted 10–45 min. The
2.5.1. Demographics form
scenarios were experiment condition-specific, which meant that all op­
A demographic survey collected background information about
erators experienced the same scenario for a particular experiment con­
operator age, experience, and level of fatigue.
dition. In all scenarios, a valve malfunction occurred during the standard
procedures following the K601 trip. The exact valve that malfunctioned
2.5.2. Workload (NASA-TLX)
was different in each scenario. Five scenarios contained valves that
The NASA-TLX (Hart, 1986; Hart and Staveland, 1988) is a subjec­
impacted vessel levels and, in one scenario, the valve controlled pres­
tive questionnaire and psychometrically validated to measure workload
sure. When the operator closed the particular valve, a malfunction
on six different constructs (mental, physical, temporal, performance,
occurred such that the valve appeared to be closed on the operator’s
effort, and frustration). When completing the NASA-TLX, the operator
display, but it remained open in the (simulated) plant (Fig. 2.). In one
placed an “X” on the continuum for each separate construct. Following
scenario, the malfunction differed slightly in that a valve that was nor­
data collection, using the scale hash marks, the researchers converted
mally closed, and appeared closed on the display, drifted to 30% open in
the operator’s “X” marks to numerical form. The scale ranged from 1
the first minute following the “K601 trip”. A description of the source
(low workload) to 20 (high workload). Thus, an X on the low end of the
alarms for each scenario is available in Table 2.
scale would be a low number, an X in the middle of the scale would be
equal to “10”, and an X on the high end of the scale would be converted
2.4. Experimental design to a higher number. Once the conversions were made for each operator,
the values for each scale were summed together. For each administra­
The study compared three types of alarm design. The three types tion of the NASA-TLX, each operator had one combined NASA-TLX score
were No-rationalization (which yielded approximately 250 alarms), that ranges from 6 to 120. The operators completed the TLX after each of
Rationalized (which yielded approximately 150 alarms), and Smart/ six simulated scenarios, which resulted in six unique TLX scores per
State-based alarms (which yielded approximately 5 alarms). These operator (one per scenario).
alarming conditions were considered separately to replicate the oper­
ating environment of our simulated environment. In this case, a smart 2.5.3. Flare release
alarm and rationalized alarm strategy cannot co-exist within the same The amount of flaring (i.e., the amount of excess chemicals released)
scenario. Thus, they were used in comparison to one another. These was recorded by the simulator’s historian function. In the actual plant,
alarm levels were designed by a subject matter expert (SME) in the after K601 trips, an application reduces furnace feed by 22% within 3
process control operations domain with over 30 years of experience who min. For consistency in the current study, the feed rate was left at that
consulted with a Ph.D. level Industrial Engineer with 30 years of expe­ level, although in a non-simulated K601 trip event, the feed rate would
rience working in process control. The SME developed each alarm level, have been reduced further by operations. The total flare released was
other than no rationalization, to replicate the processes that would occur collected by importing historian data into a Microsoft Excel file in 1-min
averages. The averages were then totaled in flare megagram per hour
Table 2 over the period of time between the first alarm onset and when the
Scenario descriptions. operator identified the anomaly.
Scenario Conditions Source alarms
2.6. Procedure
1 Rationalization W/Automation Two vessel and one valve alarm
2 Rationalization W/O Automation Two vessel pressure alarms
3 Smart Alarm W/Automation One exchanger alarm Prior to the operators’ scheduled training days, the simulation
4 No Rationalization W/Automation Two vessel and one valve alarm specialist explained that an alarm design evaluation was underway. The
5 No Rationalization W/O Automation One exchanger alarm operators were informed that, while the simulation exercises in the
6 Smart Alarm W/O Automation One vessel alarm
evaluation were part of their regular operator training, their data would

4
R.J. Simonson et al. Applied Ergonomics 100 (2022) 103670

Fig. 2. Valve malfunction process.

be depersonalized and investigated at the aggregate level and not used released was conducted to determine how the alarm and automation
as job-performance indicators for personnel assessment. Additionally, a conditions affected it. The analysis indicated that both the alarm (F(2,
partial counter-balance technique was employed such that a Latin- 20) = 8.66, p = .002, η2p = 0.46) and automation (F(1, 10) = 26.06, p <
square procedure was used to develop a counterbalanced order of con­ .001, η2p = 0.72) resulted in significantly different values of flaring such
ditions/scenarios where operators were randomly assigned to the
that the smart alarm and non-rationalized alarm conditions released the
particular order of scenarios (Goodwin and Goodwin, 2012).
least amount via the flare released, while the rationalized alarm released
Upon arriving for the training day, each operator first completed the
the most. There was no significant difference in the amount of flare
demographic questionnaire. Next, the operator performed the first of the
released between the smart alarm and non-rationalized alarm condi­
six scenarios. While the operator performed the simulated scenario, the
tions. Additionally, the condition with the safe park automation resulted
simulation specialist recorded the time the participant used to complete
in a significantly lower flare release than the condition without auto­
the task and the time to detect the anomaly.
mation (Table 4.). Further evaluation via pairwise comparisons of the
After the completion of each scenario the operator completed the
mean differences by way of t-tests of these effects and their results are
NASA-TLX questionnaire. The simulation specialist then debriefed the
presented in Table 5.
operator about the scenario, and the operator was given a short break of
about 30 min. This process repeated until the operator had completed all
six scenarios. At this time, the simulation specialist provided a full 3.2. Workload
debrief of the evaluation. The total time was approximately 8 h. As the
training was part of required training, operators were compensated with A post-hoc analysis of variance was also conducted on the dependent
their typical training day compensation. After each operator completed variable workload to determine how the alarm and automation condi­
the scenarios, the simulation specialist collected the automatically tions affected it. The analysis indicated that both the alarm (F(2, 20) =
recorded data from the simulator and transferred it to a data recording 17.43, p < .001, η2p = .64) and automation (F(1, 10) = 21.72, p = .001,
device. η2p = .69) conditions had significant differences in average reported
workload. These results indicated that the smart alarm resulted in
3. Results
significantly less reported workload than the rationalized and non-
rationalized alarms whereas no significant difference was found be­
The multivariate omnibus analysis indicated that there were large
tween rationalized and non-rationalized alarms (Table 6.). Additionally,
main effects on the dependent variables for both alarm (F (4,7) = 7.11, p
the condition with the safe park automation resulted in a significantly
= .013, η2p = .80) and automation (F (2,9) = 13.90, p = .002, η2p = .76).
lower workload than the condition without automation. Further evalu­
No interaction effects were found (F (4,7) = 2.83, p = .109, η2p = .62) ation via pairwise analyses of mean differences by way of t-tests is
(see Table 3). Although the effect size of the interaction effect was high available in Table 7.
(η2p = .62), the interaction effects power of .474 was insufficient to
detect any effect (Cohen, 1988). To better understand the intricacies of 4. Discussion
the main effects, post-hoc analyses were used to determine the indi­
vidual effects of automation and alarming condition on workload and In summary, this study investigated the effects of alarm design and
released flare amount. Correlational values between workload and automation on operator performance and workload in a high-fidelity,
performance were assessed to understand the relationship between the ethylene manufacturing simulation. The interventions in this experi­
two variables within their individual scenarios. Our results indicate that ment both significantly impacted the outcomes of the amount of mate­
scenario 1, scenario 2, and scenario 4 yielded the largest correlational rial released by flare and operator perceived workload. Specifically,
values (r = 0.458, 0.430, 0.656) respectively while scenario 3, scenario operators indicated lower perceived workload and the system released
5, and scenario 6 yielded the lowest correlational values (r = − 0.126, less material to the flare when using the safe park automation compared
0.128, − 0.229) respectively. See Fig. 3 for a graphical representation of to conditions without it. Additionally, operators reported lower
these results. perceived workload when using smart alarming versus the rationalized
or non-rationalized alarm schemas. These statistically significant re­
3.1. Flare ductions in the performance variables measured in this experiment
presents further evidence that implementing alarm management sche­
A post-hoc analysis of variance on the dependent variable flare mas, specifically smart alarming, and automation techniques to assist
human operators improves system performance.
Table 3 Consistent with prior evidence the implementation of automation
Omnibus two-way multivariate analysis of variance (MANOVA). technologies reduced workload and flare release levels regardless of the
type of alarm system used (e.g., non-rationalized, rationalized, and
DV F Hypothesis (Error) df p η2p power
smart). These results are in clear alignment with the body of literature
Alarm 7.11 4(7) .013 .80 .88 indicating that well-designed automation technology can lower human
Automation 13.90 2(9) .002 .76 .98 workload and improves performance (e.g., Endsley, 1999; Kaber and
Alarm * Automation 2.83 4(7) .109 .62 .47
Endsley, 2004). We posit that this is due to the safe-park’s direct effect

5
R.J. Simonson et al. Applied Ergonomics 100 (2022) 103670

Fig. 3. Standardized workload and flare by alarm and automation condition.

Table 4 Table 6
Flare released (Mg) Descriptive statistics by condition. Perceived workload (NASA-TLX; 0–120) Descriptive statistics by condition.
Safe-Park Automation No Automation Total Safe-Park Automation No Automation Total

M SE M SE M SE M SE M SE M SE

Smart-alarm 10.53 (3.17) 36.65 (6.49) 23.59 (4.65) Smart-alarm 34.46 (6.79) 57.09 (5.93) 45.77 (5.82)
Rationalized 20.56 (7.37) 86.68 (11.74) 53.62 (6.24) Rationalized 56.36 (5.61) 80.91 (4.05) 68.64 (2.92)
Not Rationalized 19.75 (5.16) 31.86 (8.56) 25.81 (4.78) Not Rationalized 58.36 (5.80) 73.0 (4.50) 65.68 (4.15)
Total 16.94 (2.93) 51.73 (5.21) Total 49.73 (5.12) 70.33 (3.39)

Table 5 Table 7
Flare released (Mg) pairwise comparisons. Workload pairwise comparisons.
Condition Comparisons Mean Standard p Condition Comparisons Mean Standard p
Difference Error Difference Error

Alarm Alarm
Rationalized Non- 27.81 9.21 .04 Rationalized Non- 2.96 3 1
Alarm Rationalized Alarm Rationalized
Alarm Alarm
Smart Alarm 30.03 7.95 .011 Smart Alarm 22.86 4.27 .001
Smart Alarm Non- − 2.22 6.79 1 Smart Alarm Non- − 19.91 5.10 .009
Rationalized Rationalized
Alarm Alarm

Automation Trip No Trip − 34.79 6.81 <.001 Automation Trip No Trip − 20.61 4.42 .001
Application Application Application Application

on the system wherein the automation will shut down portions of the workload, and regardless of automation, the smart alarm condition
process that are under normal operations to isolate the resulted in the lowest reported workload scores – significantly lower
out-of-control-process. Alarm management, on the other hand, is an than both the rationalized and non-rationalized alarm conditions. In
indirect solution that will only attempt to reduce the number of alarms terms of flare release, and again regardless of automation, the smart
and present potentially relevant information to the operator. Therefore, alarm condition and non-rationalized alarm condition resulted in the
the safe-park automation may assist operator decision-making and lowest amount of flare released (but were statistically similar to one
reduce operator mental processing time more effectively than alarm another). Upon inspection of the interaction between automation and
management. alarm conditions for the material lost through flare (Fig. 3.), we found
The alarm design results were mixed. First, in terms of perceived that the no-safe-park automation, rationalized alarm condition resulted

6
R.J. Simonson et al. Applied Ergonomics 100 (2022) 103670

in the highest amount of material lost via flare release. At the same time, scenarios that were similar in complexity, yet involved different valves
the no-safe-park automation, smart alarm and the no-safe park,non- (to avoid learning effects), proved to be somewhat difficult. As such,
rationalized alarm conditions were statistically insignificant from one some of the scenarios may have been more or less inherently difficult
another – an inconsistency with prior investigations on the relationships based on the challenge the operators were required to find and solve.
between rationalized alarm protocols and performance (Mahajan and The scenario in the rationalized alarm condition without automation
Surve, 2012; Adhitya et al., 2014). Additionally, comments from oper­ involved the failure of a pressure controller whereas all other scenarios
ators during the rationalized, no automation scenario indicated that in this study involved valve failures that impacted vessel level. Addi­
some operators had trouble finding certain alarms in question. This tionally, this same scenario required two displays to garner the neces­
suggests that the rationalized no-automation scenario may have been sary failure information whereas the other scenarios only required one
more difficult to complete than other scenarios. In turn, this leads to the display. Hence, the differences in this scenario as compared to the other
possibility that material lost through flare may not have been affected by five scenarios may have clouded the effect of the alarm design.
alarm rationalization techniques, rather, only by automation schema. Additionally, due to the base scenario being a plant outage, many
Importantly, when considering both alarm rationalization and auto­ possible problems were automatically eliminated. This automatic
mation schemas as well as their effects on workload and material lost elimination of problems may have reduced operator workload and, in
through flare, the data indicates that smart alarming was the most turn, the amount of time it took the operator to solve the challenges and
effective alarm state in this simulation. This is most likely due to the may have decreased variance between the conditions making it difficult
significant concatenation in alarming that the smart alarm process to detect performance differences. Further, the objective of our research
provides. Consistent with prior literature, appropriate alarm manage­ was to identify the performance and workload differences among non-
ment may provide operators with the tools needed to reduce their rationalized, rationalized, and smart alarming schemas with and
workload and increase their performance. Subsequently, future research without automation. An inherent property of the alarm rationalization
should investigate the effects of varying task difficulty on material lost schemas and automation is that the number of alarms that sound differs.
through flare based on alarm rationalization techniques. This is most While the number of alarms is not the purpose of this investigation, they
likely due to the significant concatenation in alarming that the smart are embedded within the alarming and automation conditions and thus
alarm process provides. This result indicates that consistent with prior are a possible confound within this investigation. Other possible limi­
literature, appropriate alarm management may provide operators with tations include those differences in alarm graphics (e.g., some alarm
the tools needed to reduce their workload and increase their perfor­ graphics have relatively more information displayed on them than do
mance. Subsequently, future research should investigate the effects of other alarm graphics) which may have introduced a confound that
varying task difficulty on material lost through flare based on alarm resulted in increased workload or the amount of flare released. Addi­
rationalization techniques. tionally, our data resulted in collinearity and normality challenges in
Our results regarding the correlations between workload and per­ some of our variables. Specifically, there was a lack of collinearity (r <
formance as measured by the amount of flare released are inconclusive. 0.2) between workload and flare released for scenarios three and five.
The correlational values ranged from relatively low to relatively high. We decided to include these variables in our analyses due to the
We posit that these results may indicate that the underlying mechanisms importance they have in our study. Finally, our limited sample size, due
behind a task influence the relationship between workload and perfor­ to the in-situ and applied nature of this investigation, was not optimal
mance, which may imply that similar assumptions from prior research for our MANOVA. Future research should replicate this investigation to
hold true (Cox-Fuenzalida, 2007). We found that a large dichotomy determine if similar affects are found. Further, the use of our latin-
between the correlations of workload and performance exists between square condition order procedure required either 12 or 18 participants
scenarios. Specifically, scenario 1 (rationalized alarm w/automation) to equally fulfill each randomized order. Subsequently, one order of
scenario 2 (rationalized alarm w/o automation) and scenario 4 (non-­ conditions only contained one participant instead of two.
rationalized alarm w/automation) yielded the largest correlational
values while scenario 3 (smart alarm w/automation) scenario 5 (non-­ 5. Conclusion
rationalized alarm w/o automation) and scenario 6 (smart alarm w/o
automation) yielded the lowest correlational values. Generally, sce­ The implications of this work provide insight into ways to improve
narios with rationalized alarms or automation yielded the highest cor­ alarm rationalization and automation in regard to human performance
relations while scenarios without rationalization or automation yielded in chemical processing systems. First, this is the only study to date that
the lowest except for scenario 3. However, without an experimental provides evidence that alarm strategies via rationalization, smart state-
design specifically geared towards understanding the causal effects of based schemas, and automation provide a significant improvement in
alarm rationalization and automation on the ratio of workload and performance and workload for chemical processing experts using a
performance levels, we are unable to determine if a true effect lies here. human-in-the-loop, high fidelity simulation platform. Chemical pro­
Thus, future research should explore the relationships between the cessing is complex, with modern equipment, product production, and
correlations of workload and performance with alarm rationalization refinement techniques that have created environments where operators
schemas, automation, and their interactions from a causality point of are expected to oversee and manage multifaceted and complex tasks
view. Implications for optimal levels of performance and workload may with a multitude of interacting elements. Further, the product itself is
yield a greater ratio between performance levels and workload over dangerous, particularly in fields such as ethylene development, with
longer-term periods of performance as compared to simply optimizing mishandling potentially leading to hazardous scenarios. These scenarios
alarm rationalization and automation techniques to yield the greatest require full collaboration between operator and machine to stop errors
performance or workload level independently. Additionally, under­ from propagating. While these modern systems are designed to achieve
standing the ratios of performance and workload on long-term outcomes longer and more stable periods of system homeostasis, the complexity of
is required before conclusive evidence regarding these ratios is possible. the systems means that system instability can cause a chain reaction of
every monitor and alarm directly or possibly indirectly linked to it.
4.1. Limitations Through this mechanism, when an out-of-control process occurs, hun­
dreds of alarms may be produced as a result of one malfunction
Our findings that suggest the rationalized alarm system resulted in (Strobhar, 2012). This requires that modern operators have the ability to
the highest average flare released, however, this is likely a matter of the identify which alarms are and are not necessary in order to discover the
scenarios used in this evaluation; the challenge of implementing the source of the malfunction.
simulated scenarios resulted in a few limitations. First, designing six Overall, the results of this investigation indicate that automation and

7
R.J. Simonson et al. Applied Ergonomics 100 (2022) 103670

state-based alarm systems assisted operators to resolve system abnor­ Hancock, P.A., Jagacinski, R.J., Parasuraman, R., Wickens, C.D., Wilson, G.F., Kaber, D.
B., 2013. Human-automation interaction research: past, present, and future. Ergon.
malities, failures, or otherwise adverse process events. Improving
Des 21 (2), 9–14.
operator performance by these means provides organizations the ability Hart, S.G., 1986. NASA Task Load Index (TLX). Volume 1.0. Paper and pencil package.
to enhance operators’ abilities to successfully safely and efficiently https://humansystems.arc.nasa.gov/groups/TLX/downloads/TLXScale.pdf.
complete their tasks. The rationalized alarm used in this study does not Hart, S.G., Staveland, L.E., 1988. Development of NASA-TLX (task load index): results of
empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (Eds.), Human
match the expected results formed by way of prior evidence from pre­ Mental Workload. North Holland Press., Amsterdam.
vious research. Thus, further investigations into the effects of rational­ Hollender, M., Beuthel, C., 2007. Intelligent alarming. ABB Rev. 1, 20–23.
ized alarms on operator performance should be prioritized in future Hollifield, B., 2010. Understanding and Applying the ANSI/ISA 18.2 Alarm Management
Standard. PAS Inc., Houston.
research in this domain. Human-in-the-loop simulation-based research Kaber, D.B., Endsley, M.R., 2004. The effects of level of automation and adaptive
is essential to fully understand the effects of alarm and automation automation on human performance, situation awareness and workload in a dynamic
design in process control and other domains. control task. Theor. Issues Ergon. Sci. 5 (2), 113–153.
Lee, J.D., Gore, B.F., Campbell, J.L., 1999. Display alternatives for in-vehicle warning
and sign information: message style, location, and modality. Transport. Hum.
Funding Factors 1 (4), 347–375.
Lee, J.D., Seppelt, B.D., 2009. Human factors in automation design. In: Shimon, Y.N.
(Ed.), Springer Handbook of Automation. Springer., Berlin, pp. 417–436.
Financial support for this project was from the Center of Operator Lee, J.D., Seppelt, B.D., 2006. Human factors and ergonomics in automation design. In:
Performance. Salvendy, G. (Ed.), Handbook of Human Factors and Ergonomics. Wiley., New York,
pp. 1615–1642.
Lorenz, B., Di Nocera, F., Röttger, S., Parasuraman, R., 2001. The effects of level of
Declaration of competing interest
automation on the out-of-the-loop unfamiliarity in a complex dynamic fault-
management task during simulated spaceflight operations. In: Proceedings of the
The authors declare the following financial interests/personal re­ Human Factors and Ergonomics Society Annual Meeting, 45. SAGE Publications,
lationships which may be considered as potential competing interests: Sage CA: Los Angeles, CA, pp. 44–48. No. 2.
Mahajan, S.A., Surve, V.G., 2012. Operator performance enhancement by alarm
Financial support for this project was from the Center for Operator management. In: SPE International Production and Operations Conference &
Performance. Exhibition. Society of Petroleum Engineers.
Miller, G.A., 1956. The magical number seven, plus or minus two: some limits on our
capacity for processing information. Psychol. Rev. 63 (2), 81.
Acknowledgments NIST/SEMATECH e-Handbook of Statistical Methods, 2013. 6.3.2. What are variables
control charts? https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc32.
We would like to acknowledge Dr. David Strobhar of the Center for htm#Trend.
Nochur, A., Vedam, H., Koene, J., 2001. Alarm performance metrics. IFAC Proc. Vol. 34
Operator Performance for his contributions to this project, and his (27), 203–208.
consultation regarding alarm strategies and automation strategies in the Ohio, E.P.A., 2014. Understanding the basics of gas flaring. http://epa.ohio.gov/Portals
process-control industries. /27/oilandgas/BasicsofGasFlaring.pdf.
Parasuraman, R., Sheridan, T.B., Wickens, C.D., 2000. A model for types and levels of
human interaction with automation. IEEE Trans. Syst. Man Cybern. Syst. Hum. 30
References (3), 286–297.
Popken, A., Krems, J.F., 2017. Automation and situation awareness. In: Boy, G.A. (Ed.),
Adhitya, A., Cheng, S.F., Lee, Z., Srinivasan, R., 2014. Quantifying the effectiveness of an The Handbook of Human-Machine Interaction. CRC Press., Boca Raton, FL,
alarm management system through human factors studies. Comput. Chem. Eng. 67, pp. 75–90.
1–12. Reddy, Y.J., Mehta, B.R., 2015. Alarm management systems. In: Mehta, B.R., Reddy, Y.J.
Beebe, D., Ferrer, S., Logerot, D., 2013. The connection of peak alarm rates to plant (Eds.), Industrial Process Automation Systems: Design and Implementation.
incidents and what you can do to minimize. Process Saf. Prog. 32 (1), 72–77. Elsevier., Amsterdam, pp. 569–582. https://doi.org/10.1016/B978-0-12-800939-
Brown, N., 2003. Alarm management/The EEMUA guidelines in practice. Measurement 0.00021-8.
Control 36 (4), 114–119. Riley, V., 1989. A general model of mixed-initiative human-machine systems. In:
Cohen, J., 1988. Statistical Power Analysis for the Behavioral Sciences, second ed. L. Proceedings of the Human Factors Society Annual Meeting, 33. SAGE Publications,
Erlbaum Associates, Hillsdale, N.J. Sage CA: Los Angeles, CA, pp. 124–128. No. 2.
Corys, 2015. Hydrocarbons portal. https://www.corys.com/en/hydrocarbons-portal. Rockwell Automation, 2017a. Alarm Management and Implementation. In: https://liter
Cox-Fuenzalida, L.E., 2007. Effect of workload history on task performance. Hum. ature.rockwellautomation.com/idc/groups/literature/documents/wp/proce
Factors 49 (2), 277–291. s-wp015_-en-p.pdf.
Dadashi, N., Golightly, D., Sharples, S., 2017. Seeing the woods for the trees: the problem Rockwell Automation, 2017b. Economic and Effective Alarm Management. In: htt
of information inefficiency and information overload on operator performance. ps://literature.rockwellautomation.com/idc/groups/literature/documents/wp/pro
Cognit. Technol. Work 19 (4), 561–570. ces-wp013_-en-p.pdf.
de Visser, E., Parasuraman, R., 2011. Adaptive aiding of human-robot teaming: effects of Sauer, J., Kao, C.S., Wastell, D., Nickel, P., 2011. Explicit control of adaptive automation
imperfect automation on performance, trust, and workload. J Cognitive Eng. under different levels of environmental stress. Ergonomics 54 (8), 755–766.
Decision Making 5 (2), 209–231. Siemens, 2018. Alarm Management in Simatic Pcs 7 – Smart Alarm Hiding. Chicago, IL.
Emam, E.A., 2015. Gas flaring in industry: an overview. Petroleum & coal 57 (5). https://cache.industry.siemens.com/dl/files/984/55699984/att_966468
Endsley, M.R., 1987. The application of human factors to the development of expert /v2/55699984_Alarm_Hiding_DOC_PCS7_V90SP1_en.pdf.
systems for advanced cockpits. In: Proceedings of the Human Factors Society Annual Sheridan, T.B., Verplank, W.L., 1978. Human and Computer Control of Undersea
Meeting, 31. SAGE Publications, Sage CA: Los Angeles, CA, pp. 1388–1392. No. 12. Teleoperators. Massachusetts Inst of Tech Cambridge Man-Machine Systems Lab.
Endsley, M.R., 1999. Level of automation effects on performance, situation awareness https://apps.dtic.mil/dtic/tr/fulltext/u2/a057655.pdf.
and workload in a dynamic control task. Ergonomics 42 (3), 462–492. Stanton, N.A., Salmon, P., Jenkins, D., Walker, G., 2009. Human Factors in the Design
Endsley, M.R., Kiris, E.O., 1995. The out-of-the-loop performance problem and level of and Evaluation of Central Control Room Operations. CRC Press., Boca Raton, FL.
control in automation. Hum. Factors 37 (2), 381–394. Strobhar, D.A., 2012. Human Factors in Process Plant Operation. Momentum Press,
Fitts, P.M., 1951. Human Engineering for an Effective Air Navigation and Traffic Control pp. 3–5.
System. National Research Council, Washington, DC. Veland, O., Kaarstad, M., Seim, L.A., Foerdestroemmen, N.T., 2001. Useful and Useable
Goodwin, C.J., Goodwin, K.A., 2012. Methodological control in experimental research. Alarm Systems: Recommended Properties (No. HWR–679). Institutt for
In: Research in Psychology: Methods and Design. Wiley., New Jersey, pp. 183–211. Energiteknikk.
Guerlain, S., Jamieson, G.A., Bullemer, P., Blair, R., 2002. The MPC elucidator: a case
study in the design for human-automation interaction. IEEE Trans. Syst. Man
Cybern. Syst. Hum. 32 (1), 25–40.