Defensive Flying for Pilots: An Introduction to Threat and Error

Management
Ashleigh Merritt, Ph.D. & James Klinect, Ph.D.
The University of Texas Human Factors Research Project1
The LOSA Collaborative
December 12, 2006
The easiest way to understand Threat and Error Management (TEM) is to liken it to
defensive driving for a motorist. The purpose of defensive driving is not to teach people how
to drive a vehicle (e.g., how to shift a manual transmission) but to emphasize driving
techniques that people can use to minimize safety risks (e.g., techniques to control rear-wheel
skids). Similarly, TEM does not teach pilots how to technically fly an airplane; instead, it
promotes a proactive philosophy and provides techniques for maximizing safety margins
despite the complexity of ones flying environment. In this sense, TEM training can be
framed as defensive flying for pilots.
TEM proposes that threats (such as adverse weather), errors (such as a pilot selecting
a wrong automation mode), and undesired aircraft states (such as an altitude deviation) are
everyday events that flight crews must manage to maintain safety. Therefore, flight crews
that successfully manage these events regardless of occurrence are assumed to increase their
potential for maintaining adequate safety margins.

It is this notion that provides the

overarching objective of TEMto provide the best possible support for flight crews in
managing threats, errors, and undesired aircraft states.
This paper provides an introductory orientation to TEM via a discussion of origins,
definitions, and techniques. We will show how TEM was initially developed to help
observers analyse activity in the cockpit and how it has since grown to become an
organizational safety management tool used in training, incident reporting, and accident and
incident analysis. TEM concepts are further explained using real-world examples and
statistics taken from the LOSA Archive, which currently contains more than 5500 TEM-

1
The University of Texas Human Factors Research Project, directed by Dr. Robert Helmreich, is funded by a
research grant from the Federal Aviation Administration, AAR-100, Human Factors Division. For more
information, contact James Klinect at [email protected].

based observations from 28 commercial airlines in over 14 countries around the world.2 The
final section, TEM tools and techniques, highlights the practical, proactive nature of TEM
and its relevance for all pilots.

Origin and Development of TEM

The origin of TEM is inextricably tied to the origin of Line Operations Safety Audits
(LOSA). It began with a simple question: Do the concepts taught in training transfer to
normal, everyday flight operations? The question prompted a partnership between The
University of Texas Human Factors Research Project (UT) and Delta Airlines in 1994 to
develop a line audit methodology utilizing jump-seat observations on regularly scheduled
flights. All parties realized that in order for the audit to work, i.e., to really see what
happened on the line, there had to be a guarantee of confidentiality with no regulatory or
organizational jeopardy for the crews that were observed. Crews had to believe there would
be no individual repercussions; otherwise, they would revert to their best angel
performance when being observed and the audit would uncover nothing more than what was
learned from line check or training data.
The first observation form was designed by the UT researchers to evaluate Crew
Resource Management (CRM) behaviours. The form was then expanded to address error and
its management. As well as type of error committed, the form prompted observers to note
who caused the error, the response to the error (i.e., whether the error was detected and by
whom), and the outcome of the error. Knowing an error occurred without really knowing the
conditions under which it occurred seemed to tell only part of the story. Hence, the
researchers developed and included the concepts of threat and threat management in the
observation form to capture the full operational complexity of a flight.
The first full TEM-based LOSA was conducted at Continental Airlines in 1996. Data
from the observation forms were aggregated to develop an airline profile. As well as the
original CRM indicators such as leadership, communication, and monitoring/cross-checking,
the TEM organizational profile highlighted the most frequent threats, threats that were wellmanaged versus more problematic threats (i.e., those that were mismanaged at higher rates
than other threats), the most common errors, the least versus more problematic errors, and the

LOSA stands for Line Operations Safety Audit. See Appendix A for details on the LOSA Archive and how it
was built, including a list of airlines with observations in the Archive.

rate of Undesired Aircraft States, including unstable approaches. Among other things, the
airline learned that it had issues with its checklists. It also realized there were no clear
guidelines on when to execute a missed approach, which could explain the rate of unstable
approaches. With a data-driven report that highlighted operational strengths and weaknesses,
the airline set up cross-departmental committees from Flight Operations, Ground Operations,
Training, and the Safety Department to work on solutions.
The company also instigated a one-day TEM training course for all its pilots.
Trainers introduced the concepts of Threat and Error and then debriefed the LOSA findings.
As a result, pilots were able to see a different perspective of safety performance at their
airline as reflected in organizational threat and error prevalence and management rates. The
pilots responded positively, analysing the data for reasons, and using what they learned to
proactively enhance their own performance.
Using the 1996 LOSA results as a baseline, Continental conducted a follow-up LOSA
in 2000. To quote Captain Don Gunther, Senior Director of Safety & Regulatory Compliance
at Continental Airlines:
The 2000 LOSA, when compared to the results of 1996, showed the pilots
had not only accepted the principles of error management but incorporated
them into everyday operations. LOSA 2000 showed a sizeable improvement
in the areas of checklist usage, a 70 percent reduction in non-conforming
approaches (i.e., those not meeting stabilized approach criteria), and an
increase in overall crew performance. It could be said that Continental had
taken a turn in the right direction.
Based on the success at Continental as well as other LOSA carriers, the International
Civil Aviation Organization (ICAO) made LOSA a central focus of its Flight Safety and
Human Factors Program and endorsed it as an industry best practice for normal operations
monitoring (ICAO LOSA Manual, Doc 9803). The Federal Aviation Administration (FAA)
also endorses LOSA as one of its voluntary safety programs (FAA Advisory Circular 12090). As a result, TEM and LOSA are now recognised world-wide.

TEM: Definitions, Examples, and Quizzes

The Threat and Error Management (TEM) framework focuses simultaneously on the
operating environment and the humans working in that environment. Because the framework
captures performance in its natural or normal operating context, the resulting description is
realistic, dynamic, and holistic. Because the TEM taxonomy can also quantify the specifics of
the environment and the effectiveness of performance in that environment, the results are also
highly diagnostic.

Threats and their Management

Pilots have to manage various complexities in the operating environment on a typical day
of flying. In TEM, such complexities are known as threats.

Threat Definition
Threats are defined as events or errors that:

occur outside the influence of the flight crew (i.e., not caused by the crew);

increase the operational complexity of a flight; and

require crew attention and management if safety margins are to be maintained.

Using this definition, a threat can be high terrain, icing conditions, an aircraft
malfunction (e.g., inoperative thrust reverser), or other peoples errors, such as an inaccurate
recording of a fuel load by a dispatcher. All these events occur independently of the flight
crew, yet they add to the crews workload and need to be managed. Sometimes they can be
managed discreetly and sometimes they interact with one another further complicating the
necessary management. In commercial airlines, threats can be divided into two categories:
environmental threats, which are outside the airlines direct control, such as weather and
ATC; and airline threats, which originate within flight operations, such as aircraft
malfunctions and ground problems. The table below shows the various threat types with
examples.

Threat Types with Examples

Environmental Threats

Adverse Weather

Airport

ATC

Environmental Operational
Pressure

Airline Threats

Aircraft

Airline Operational Pressure

Cabin

Dispatch/Paperwork

Examples

Thunderstorms, turbulence, poor visibility, wind shear, icing conditions, IMC

Poor signage, faint markings, runway/taxiway closures, INOP navigational

aids, poor braking action, contaminated runways/taxiways
Tough-to-meet clearances/restrictions, reroutes, language difficulties,
controller errors

Terrain, traffic, TCAS TA / RA, radio congestion

Examples
Systems, engines, flight controls, or automation anomalies or malfunctions;
MEL items with operational implications; other aircraft threats requiring flight
crew attention
On-time performance pressure, delays, late arriving aircraft or flight crew

Cabin events, flight attendant errors, distractions, interruptions

Load sheet errors, crew scheduling events, late paperwork, changes or errors

Ground/Ramp

Aircraft loading events, fuelling errors, agent interruptions, improper ground

support, de-icing

Ground Maintenance

Aircraft repairs on ground, maintenance log problems, maintenance errors

Manuals/Charts

Missing information or documentation errors

Threat management can be broadly defined as how crews anticipate and/or respond to
threats. A mismanaged threat is defined as a threat that is linked to or induces flight crew
error. Some of the common tools and techniques used in commercial aviation to manage
threats and prevent crew errors include reading weather advisories, turning weather radar on
early, thorough walk-arounds during predeparture, correct use of procedures to diagnose
unexpected aircraft malfunctions, briefing an alternate runway in case of a late runway
change, briefing cabin crew as to acceptable times and reasons for interruptions, and loading
extra fuel when the destination airport is in question due to poor weather or restricted access.
Just how common are threats and when do they occur? Take the quiz below to find out.

Threat Management Quiz

Test your knowledge of threats and their management by circling your best guess to the following questions about
findings from the LOSA Archive of more than 4500 observations across 25 airlines. Correct answers with
discussion will be provided at the end of the quiz.

1.

2.

3.

4.

5.

On average, how many threats per flight (regularly scheduled, normal operations) are encountered by
flight crews in the LOSA Archive?
A) One threat every 2-3 flights

C) 1-3 threats per flight

B) One threat per flight

D) 4-6 threats per flight

In what phase of flight do most threats occur in the LOSA Archive?

A) Predeparture/Taxi-out

C) Cruise

B) Takeoff/Climb

D) Descent/Approach/Land

What are the most frequently encountered threats by flight crews in the LOSA Archive?
A) Adverse weather (e.g., thunderstorms)

C) Aircraft (e.g., malfunctions / anomalies)

B) ATC (e.g., challenging clearances)

D) Airport (e.g., poor signage/construction)

What percent of threats are successfully managed by flight crews in the LOSA Archive? (i.e.,
percentage of threats not contributing to a flight crew error)
A) 95-100%

C) 75-85%

B) 85-95%

D) Less than 75%

Of all threats encountered by flight crews in the LOSA Archive, which are the most problematic?
A) Adverse weather (e.g., thunderstorms)

C) Aircraft (e.g., malfunctions / anomalies)

B) ATC (e.g., challenging clearances)

D) Airport (e.g., poor signage/construction)

Threat Management Quiz Answers and Discussion

1. The correct answer is (D). Based on the last 25 LOSAs (over 4500 flights in total) in the LOSA Archive, the
typical flight (regularly scheduled, normal operations) encounters an average 4.2 threats per flight. Of those, three
are likely to be Environmental threats and one is likely to be an Airline threat. Only 3% of flights encounter no
threats whatsoever, while 17% of flights encounter seven or more threats per flight. In other words, multiple
threats are the standard and should be considered as such in every flight.
2. The correct answer is (A). Overall, about 40% of all threats occur during Predeparture/Taxi-out and 30% occur
during Descent/Approach/Land. Different types of threats are more prevalent during different phases of flight. For
Environmental threats (weather, ATC, terrain, traffic, airport conditions), the busiest phase of flight is
Descent/Approach/Land, while for Airline threats, the busiest phase is Predeparture/Taxi-out. In percentage
terms, 43% of all Environmental threats occur during Descent/Approach/Land, while 73% of all Airline threats
occur during Predeparture/Taxi-out.
3. The correct answer is (A or B). With 4500 flights having an average of 4.2 threats per flight, there are 19,000
logged threats in the LOSA Archive. So which are the most common? Actually, Adverse Weather and ATC both
account for about one quarter of all observed threats, followed by Aircraft Threats (about 13% of all observed
threats) and Airport Conditions (about 7% of all observed threats).
4. The correct answer is (B). 85-95% of all threats are successfully managed. The average across the Archive is
90%. Put another way, about one-tenth of all threats are mismanaged by the crews, leading to some form of crew
error.
5. The correct answer is (B). Mismanagement rates are actually very close for the top three offenders. Thirteen
percent of Aircraft threats, 12% of ATC threats, and 11% of Adverse Weather threats are typically mismanaged.
However, when you combine these mismanagement rates with the frequency with which different threats occur,
ATC threats emerge as the most problematic threat. In particular, challenging clearances and late changes from
ATC are the most problematic of all threats for flight crews.

Errors and their Management

From the TEM perspective, error is a crew action or inaction that leads to a deviation
from crew or organizational intentions or expectations. Put simply, threats come at the
crew, while errors come from the crew.

Flight crew errors can be the result of a

momentary slip or lapse, or induced by an expected or unexpected threat. For example, a late
runway change might induce a procedural shortcut that results in further error, just as a gate
agent interruption could distract the flight crew from completing a checklist, causing them to
miss an incorrect flaps setting for takeoff. Other errors are more deliberate. Known as
intentional noncompliance errors in the TEM taxonomy, these errors are often proven
shortcuts used by flight crews to increase operational efficiency even thought they are in
violation of Standard Operating Procedures. High rates of noncompliance at an airline can
often indicate systemic over-procedualization.

Error Definition
Errors are defined as flight crew actions or inactions that:

lead to a deviation from crew or organizational intentions or expectations;

reduce safety margins; and

increase the probability of adverse operational events on the ground or during flight.

Flight crew errors can be divided into three types: aircraft handling, procedural and
communication errors. Aircraft handling errors are those deviations associated with the
direction, speed and configuration of the aircraft. They can involve automation errors, such
as dialling an incorrect altitude, or hand-flying errors, such as getting too fast and high during
an approach. Procedural errors are flight crew deviations from regulations, flight manual
requirements or airline standard operating procedures. Lastly, communication errors involve
a miscommunication between the pilots, or between the crew and external agents such as
ATC controllers, flight attendants, and ground personnel. The table below shows the various
error types with examples.

Error Types with Examples

Aircraft Handling Errors

Automation

Flight Control

Ground Navigation

Manual Flying

Systems/Radio/Instruments

Examples

Incorrect altitude, speed, heading, autothrottle settings, mode executed, or

entries

Incorrect flaps, speed brake, autobrake, thrust reverser or power settings

Attempting to turn down wrong taxiway/runway

Missed taxiway/runway/gate
Hand flying vertical, lateral, or speed deviations
Missed runway/taxiway failure to hold short, or taxi above speed limit

Incorrect pack, altimeter, fuel switch or radio frequency settings

Procedural Errors

Examples

Briefings

Missed items in the brief, omitted departure, takeoff, approach, or handover

briefing

Callouts

Omitted takeoff, descent, or approach callouts

Checklist

Documentation

Pilot Flying (PF)/Pilot Not

Flying (PNF) Duty

SOP Cross-verification

Other Procedural

Communication Errors

Crew to External

Pilot to Pilot

Performed checklist from memory or omitted checklist

Missed items, wrong challenge and response, performed late or at wrong time
Wrong weight and balance, fuel information, ATIS, or clearance recorded
Misinterpreted items on paperwork

PF makes own automation changes, PNF doing PF duties, PF doing PNF duties

Intentional and unintentional failure to cross-verify automation inputs

Other deviations from government regulations, flight manual requirements or

standard operating procedures
Examples
Missed calls, misinterpretation of instructions, or incorrect read-backs to ATC
Wrong clearance, taxiway, gate or runway communicated

Within-crew miscommunication or misinterpretation

Error management is now recognized as an inevitable part of learning, adaptation, and

skill maintenance; hence, a primary driving force behind TEM is to understand what types of
errors are made under what circumstances (i.e., the presence or absence of which threats) and
how crews respond in those situations. For example, do crews detect and recover the error
quickly, do they acknowledge the error but do nothing, perhaps because they believe it is
inconsequential or will be trapped later, or do they only see the error when it escalates to a
more serious undesired aircraft state? This is the heart of error management: detecting and
correcting errors. However, approximately 45% of the observed errors in the LOSA Archive
were errors that went undetected or were not responded to by the flight crew, which gives
credence to an important point for effective error management: An error that is not detected
cannot be managed.
An error that is detected and effectively managed has no adverse impact on the flight.
On the other hand, a mismanaged error reduces safety margins by linking to or inducing
additional error or an undesired aircraft state.3 Just how common are mismanaged errors and
when do they occur? The LOSA Archive provides some insight, as shown in the quiz below.

Undesired Aircraft State (UAS): A flight-crew-induced aircraft state that clearly reduces safety margins; a safetycompromising situation that results from ineffective error management. Discussed in next section.

10

Error Management Quiz

Test your knowledge of flight crew errors and their management by circling your best guess to the following
questions. As with the Threat Management Quiz, correct answers with discussion will be provided at the end of
the quiz.
1.

Of flights in the LOSA Archive, how common is flight crew error?

A) Approximately 5% of flights have some form of observable crew error
B) Approximately 50% of flights have some form of observable crew error
C) Approximately 80% of flights have some form of observable crew error
D) All LOSA flights (100%) have at least one observable crew error

2.

3.

In what phase of flight do most flight crew errors occur in the LOSA Archive? When do the mismanaged
errors occur? (Hint: The answer is the same phase of flight for both questions)
A) Predeparture/Taxi-out

C) Descent/Approach/Land

B) Takeoff/Climb

D) Taxi-in/Park

What are the most frequently committed flight crew errors in the LOSA Archive?
A) Aircraft Handling (e.g., wrong automation setting)
B) Procedural (e.g., omitted callout)
C) Communication (e.g., incorrect ATC readback)

4.

5.

6.

What are the most common procedural errors observed in the LOSA Archive?
A) Briefing

C) Callout

B) SOP Cross-verification

D) Checklist

What percentage of errors are mismanaged by flight crews in the LOSA Archive (i.e., percentage of
errors linking to an additional error or undesired aircraft state)
A) 20-30%

C) 40-50%

B) 30-40%

D) More than 50%

What are the most frequently mismanaged flight crew errors in the LOSA Archive?
A) Manual Handling/Flight Control

C) System/Instrument/Radio

B) Automation

D) Checklist

11

Error Management Quiz Answers and Discussion

1. The correct answer is (C). Based on the last 25 LOSAs (over 4500 flights in total) in the LOSA Archive, about
80% of flights have one or more errors the average is about three errors per flight. Twenty percent of flights
have no observable error.
2. The correct answer is (C). The busiest phase of flight for errors is Descent/Approach/Land. About 40% of all
observed errors occur during this phase. Another 30% of errors occur during Predeparture/Taxi-out when crews
are preparing the flight.

If you look at the sub-set of errors that are mismanaged, then the rate for

Descent/Approach/Land jumps to 55%. Therefore, the most problematic phase of flight where more errors, and
more mismanaged errors, are likely to occur is Descent/Approach/Land. This likely makes intuitive senseerrors
on the ground arent as difficult to manage as errors coming down.
3. The correct answer is (B). About one-half of all observed errors are Procedural errors, one-third are Aircraft
Handling, and one-sixth are Communication errors. However, this ratio changes dramatically for mismanaged
errors. Procedural errors make up half of all errors, but a little less than one-quarter of the mismanaged errors.
Three-quarters of all mismanaged errors are Aircraft Handling errors, with Communication errors comprising the
remaining few percent.
4. The correct answer is (D). Checklist errors are the most common procedural error, followed closely by Callout
and SOP cross-verification errors. Briefing errors are less common.
5. The correct answer is (A). About 25% of all errors are mismanaged6% of all errors lead to additional error
and 19% result directly in an undesired aircraft state.
6. The correct answer is (A). Manual handling/flight control errors make up 36% of all mismanaged errors.
Automation and System/Instrument/Radio errors each make up 16% of the mismanaged errors. Checklist errors
make up 5% of the mismanaged errors; Crew-ATC communication errors make up 3% of the mismanaged errors.

12

Undesired Aircraft States and their Management

Unfortunately, not all errors are well managed. Sometimes they lead to another error
or a safety-compromising event called an undesired aircraft state (UAS).

Undesired Aircraft State Definition

An undesired aircraft state (UAS) is defined as a position, speed, attitude, or
configuration of an aircraft that:

results from flight crew error, actions, or inaction; and

clearly reduces safety margins

In other words, a UAS is a safety-compromising state that results from ineffective

error management. Examples include unstable approaches, lateral deviations from track, firm
landings, and proceeding towards the wrong taxiway/runway. Events such as malfunctions or
ATC controller errors can also place the aircraft in a compromised position; however, in the
TEM taxonomy, these events are considered threats as they are not the result of actions by the
flight crew.
UAS Types with Examples
UAS Types

Examples
Vertical, lateral or speed deviations
Unnecessary weather penetration

Aircraft Handling

Unstable approach
Long, floated, firm or off-centreline landings
Runway/taxiway incursions

Ground Navigation

Wrong taxiway, ramp, gate, or hold spot

Taxi above speed limit

Incorrect Aircraft Configuration

Automation, engine, flight control, systems, or weight/balance events

As with errors, UASs can be managed effectively, returning the aircraft to optimally
safe flight, or mismanaged, leading to an additional error, undesired aircraft state, or worse,
an incident, or accident. The last quiz sheds light on the prevalence and mismanagement of
undesired aircraft states in the LOSA Archive.

13

Undesired Aircraft State Management Quiz

Test your knowledge of undesired aircraft states and their management by circling your best guess to the
following questions. As with the previous quizzes, correct answers with discussion will be provided at the end of
the quiz.
1.

Of flights in the LOSA Archive, how common are undesired aircraft states (UAS)?
A) Less than 1% of flights have a UAS
B) 15% of flights have a UAS
C) 35% of flights have a UAS
D) 50% of flights have a UAS

2.

What are the most frequent UASs observed in the LOSA Archive?
A) Incorrect systems configurations (e.g., wrong anti-ice setting in icing conditions)
B) Speed deviations
C) Lateral and vertical deviations
D) Incorrect automation configurations (e.g., wrong altitude dialled after cross-check)

3.

How common are unstable approaches in the LOSA Archive and how often do they result in a missed
approach?
A) Less than 1% of flights have an unstable approach; of those, 95% result in a missed approach
B) 5% of flights have an unstable approach; of those, 5% result in a missed approach
C) More than 15% of flights have an unstable approach; of those, 50% result in a missed approach

4.

How many UASs in the LOSA Archive can be linked back, via mismanaged crew error, to a
mismanaged threat?
A) Virtually all UASs come about because of a threat that was mismanaged (95-100%)
B) About 70% of all UASs are linked to a mismanaged threat; the rest emerge from
spontaneous crew errors that were mismanaged (spontaneous = not linked to a threat)
C) About 30% of all UASs are linked to a mismanaged threat.

14

Undesired Aircraft States Quiz Answers and Discussion

1. The correct answer is (C). Despite being the safest form of transport, fully one-third of all flights in the LOSA
Archive have an undesired aircraft state. Numbers such as these remind us there is still room for improvement!
2. The correct answer is (A). Almost 20% of all UASs involve an incorrect aircraft system configuration (they
occur on approximately 9% of flights). Speed deviations are next at 16%, followed by lateral/vertical deviations
and incorrect automation configuration (each comprises about 13% of all UASs). These UAS types each occur on
approximately 7% of flights.
3. The correct answer is (B). In regularly scheduled, normal operations, 5% of flights involve an unstable
approach. What is disconcerting is that only 5% of those unstable approaches result in a go-around, meaning the
vast majority of crews decide to continue with the landing, even though they know they are not within specified
parameters. Are they choosing to continue the approach because of operational pressure (wanting to save time
and fuel), poor airmanship, or foolish bravado? Perhaps some of all three, what do you think?
4. The correct answer is (C). About 30% of all UASs occur as part of a chain of events that starts with a threat
that is not managed well and leads to a crew error, which in turn is mismanaged to a UAS. An example would be
an Airport Conditions threat such as poor or faded signage (threat) that confuses the crew, leading them to turn
down the wrong runway (error), which results in a runway incursion (UAS).

TEM Tools & Techniques

The principles of TEM are not new to aviation. In fact, Orville and Wilbur Wright no
doubt practiced threat and error management when they took their first controlled flight with
the Wright Flyer in 1903. Since then, various tools and techniques have been developed over
the past century to help flight crews manage threats, errors, and undesired aircraft states.
Some toolsthe hard safeguardsare associated with aircraft design, and include
automated systems, instrument displays, and aircraft warnings.

The Traffic Collision

Avoidance System (TCAS), which provides flight crews with visual and audio warnings of
nearby airplanes to prevent midair collisions, is a good example of a hard TEM safeguard.
Even with the best designed equipment however, these hard safeguards are not enough to
ensure effective TEM performance.

15

Other toolsthe soft safeguardsare very common in aviation (and other high-risk
industries). They include regulations, standard operating procedures, and checklists to direct
pilots and maintain equipment; and licensing standards, checks, and training to maintain
proficiency. With the hard and soft safeguards in place, the last line of defence against threat,
error, and undesired aircraft states, is still, ultimately, the flight crew. Checklists only work if
flight crews use them; the autopilot only works when engaged in the correct mode.
Therefore, TEM tools work best when pilots adopt TEM techniques.
The TEM philosophy stresses three basic concepts: anticipation, recognition, and
recovery. The key to anticipation is accepting that while something is likely to go wrong, you
cant know exactly what it will be or when it will happen. Hence, a chronic unease reinforces
the vigilance that is necessary in all safety-critical professions. Anticipation builds vigilance,
and vigilance is the key to recognizing adverse events and error. Logically, recognition leads
to recovery. In some cases, particularly when an error escalates to an undesired aircraft state,
recovering adequate safety margins is the first line of action: Recover first, analyse the causes
later. For example, a crew enters a Flight Management System (FMS) approach to runway
26L; however, they mistakenly enter data for 26R. Furthermore, the error is not detected by
the flight crew on a SOP required cross-verification. Once the flight crew executes the
incorrect entry and the airplane starts flying on a profile to the wrong runway, the flight is
considered to be in an undesired aircraft state. At this point, the crew can either analyze
whats wrong with the automation and fix the problem or save valuable time by simply
disconnecting the autopilot and hand-flying the approach to the correct runway. The latter
option is more effective from the TEM perspective because it focuses effort on recovering
from the undesired aircraft state rather than analyzing its causes.
While hard and soft safeguards help support pilots to best anticipate, recognize
and recover from threats, errors, and undesired aircraft states, there is arguably no better way
to manage these events in multi-pilot cockpits than through effective crew coordination.
Many of the best practices advocated by Crew Resource Management (CRM) can be
considered TEM countermeasures.
Planning

countermeasuresplanning,

preparation,

briefings,

contingency

managementare essential for managing anticipated and unexpected threats.

16

 Execution countermeasuresmonitor/cross-check, taxiway/runway management,

workload and automation managementare essential for error detection and error
response.
Review/Modify countermeasuresevaluation of plans, inquiryare essential for
managing the changing conditions of a flight, such as undesired aircraft states.
Initial research in the LOSA Archive has supported links between TEM and CRM.
For example, crews that develop contingency management plans, such as proactively
discussing strategies for anticipated threats, tend to have fewer mismanaged threats; crews
that exhibit good monitoring and cross-checking usually commit fewer errors and have fewer
mismanaged errors; and finally, crews that exhibit strong leadership, inquiry, workload
management are typically observed to have fewer mismanaged errors and undesired aircraft
states than other crews.

Conclusion: Applications of TEM

TEM is both a philosophy of safety and a practical set of techniques. Originally
designed to simultaneously capture performance and the context in which it occurs, TEM has
demonstrated its usefulness in many settings.
Training: The International Civil Aviation Organization (ICAO) has introduced a
standard making TEM training mandatory for airline flight crews engaged in international
operations. TEM training must now be delivered during initial as well as during recurrent
training. ICAO has also introduced standards making TEM training mandatory for licensing
and training requirements of private and commercial pilots and air traffic controllers. In order
to support these standards, ICAO is continually developing guidance material on TEM which
reflects and is aligned with the concepts discussed in this paper (Human Factors Training
Manual, Procedures for Air Navigation Services, Training, PANS/TRG, and An introduction
to TEM in ATC). In addition, the Australian Transport Safety Bureau and Australian Civil
Aviation Safety Authority are facilitating TEM training courses for pilots.
Line Operations Safety Audits (LOSA): Considered a best practice for normal
operations monitoring and aviation safety by both ICAO and the FAA, TEM-based LOSAs
continue to provide valuable diagnostic information about an airlines safety strengths and
vulnerabilities.

17

Incident Reporting: Several US airlines now use TEM as the conceptual structure for
their incident reporting systems. Reporting forms prompt pilots to report the threats that were
present, the errors they may have made, how the event was managed, and how the event may
have been avoided or handled better. Even pilots who have not had training in TEM are able
to complete the reporting form, a fact that speaks to the intuitive nature of the TEM
framework.
Incident and Accident Analysis: The International Air Transport Association (IATA)
Safety Committee adopted the TEM model as an analysis framework for its Incident Review
Meetings, based on its ease of use and utility of the extracted data. IATA has also created the
Integrated Threat Analysis Task Force (ITATF). This group analyses data from accidents, incidents,
and normal operations using TEM as the common framework. By selecting specific scenarios, for
example, runway excursions from the incident and accident databases, and precursors to runway
excursions from the LOSA Archive, it is possible to provide a more complete picture of safety issues
within the aviation system.

Other Aviation Settings: Studies are currently underway to adapt TEM to Air Traffic
Control, Flight Dispatch, and Ramp. Of interest, the first ATC trials, called the Normal
Operations Safety Survey (NOSS), were conducted under ICAO sponsorship in Australia,
Canada, and New Zealand, and were well-received. The ICAO sponsored NOSS manual
explaining how to conduct normal operations monitoring in Air Traffic Control, will be
available in 2007.
TEM has proved its utility in many safety management applications. As organizations
and individuals continue to adopt TEM as a way to understand and enhance their
performance, we hope that you too will see the utility of the TEM framework and find ways
to incorporate TEM techniques into your own personal philosophy of safety.

18

Appendix A: The LOSA Archive

The LOSA Archive is a database containing observers narratives and coded
observations from all the airlines that have conducted a Line Operations Safety Audit
(LOSA) with the LOSA Collaborative. Because of the stringent quality assurance process
(see below), results from different airlines can be pooled to derive industry averages. The
LOSA Archive can also benchmark an airlines performance against other airlines, providing
a multi-airline context for understanding an airlines strengths and weaknesses.
The statistics cited in this paper are based on 4,532 observations taken from the 25
most recent LOSAs (2002-2006). The data generated by those observations include 19,053
observed threats, 13,675 errors, and 2,589 Undesired Aircraft States. The LOSA Archive
currently contains observations from the following airlines.

The LOSA Archive (2002-2006)

AeroMexico (Mexico)

Continental Express (USA)

Regional Express (Australia)

Air New Zealand

Continental Micronesia

SilkAir (Singapore)

Air Transat (Canada)

Delta Air Lines (USA)

Singapore Airlines

Alaska Airlines (USA)

EVA Air / UNI Air (Taiwan)

TACA International (S America)

Asiana Airlines (Korea)

Frontier Airlines (USA)

TACA Peru (S America)

Braathens ASA (Norway)

LACSA (Central America)

US Airways (USA)

Cathay Pacific (Hong Kong)

Malaysia Airlines

WestJet (Canada)

China Airlines (Taiwan)

Mt Cook Airlines (New Zealand)

Continental Airlines (USA)

Qantas (Australia)

LOSA Quality Assurance Process

To ensure successful implementation, airlines are required to participate in a five-part LOSA
quality assurance process.
1. An agreement is reached between airline management and the pilots association. This
agreement ensures that all data will be de-identified, confidential, and sent directly to

19

the LOSA Collaborative for analysis. It also states that once the LOSA results are
presented, both parties have an obligation to use the data to improve safety.
2. The airline is assisted in selecting a diverse and motivated group of observers. A
typical observer team will have representatives from a number of different airline
departments, such as flight operations, training, safety, and the flight crew association.
3. The observers receive five days of training in the Threat and Error Management
framework, the observation methodology, and the LOSA software tool, which
organizes data input. The LOSA Collaborative software also provides data security
through automatic encryption. After the initial observer training, observers conduct at
least two sample observations and then reconvene for recalibration sessions. During
this time, observers are given one-on-one feedback on the quality of their
observations and certified to continue as observers on the project. The observer
training and recalibration are considered essential for a standardized LOSA dataset.
Subsequent observations are typically conducted over the next four to eight weeks.
4. When the encrypted observations are sent to the LOSA Collaborative, analysts read
the observers flight narratives and check that every threat and error has been coded
accurately. This data integrity check ensures the airlines data are of the same
standard and quality as other airlines in the LOSA Archive.
5. Once the initial data integrity check is complete, airline representatives who are fleet
experts attend a data-cleaning roundtable with the LOSA Collaborative analysts.
Together they review the data against the airlines procedures, manuals, and policies
to ensure that events and errors have been correctly coded. After the roundtable is
completed, airline representatives are required to sign off on the data set as being an
accurate rendering of threats and errors. Only then does analysis for the final report
begin.

20