Ar 2012 023 - Final
Ar 2012 023 - Final
Ar 2012 023 - Final
experience
andtitle
performance
Insert
document
in an airline environment
Location | Date
Researcn
Investigation
Authors:
Released in accordance with section 25 of the Transport Safety Investigation Act 2003
Publishing information
Published by:
Postal address:
Office:
Telephone:
Facsimile:
Email:
Internet:
Change
Date
Safety summary
Why have we done this report
Significant debate has occurred within the aviation industry regarding the issues of pilot training
and experience, particularly with regard to the introduction of new pilot training programs that are
focused on training cadet pilots. The main concern being presented by some sectors of the
industry that are not in favour of these concepts is that these low-hour co-pilots are not as
competent as their high-hour peers.
The ATSB gathered data from three airlines to explore the issue of pilot performance as a function
of both flight hour experience, and entry pathway. Entry pathway analysis compared cadet pilots
(who generally had not accumulated prior flight hours or experience) to those pilots who entered
an airline after accumulating flight hours in other areas of the aviation industry.
Data were collected on a number of metrics from simulator check flights, which covered nonnormal operations, and line checks, which covered normal day-to-day flight operations.
Safety message
The evidence in this report indicates that the cadet pathway for low-hour pilots is a valid option for
airlines. There was no evidence to indicate that cadets or low-hour pilots within the airlines studied
were any less competent or proficient than their direct entry and high-hour peers.
Contents
Context ......................................................................................................................................1
Literature review ......................................................................................................................3
General Aviation pilots
3
Airline pilots
4
Methodology ............................................................................................................................6
Data sources
6
Design and procedure
6
Data sources
6
Independent variables
7
Dependent variables
8
Statistical analysis
10
Results ................................................................................................................................... 11
Airline A
11
Simulator proficiency check
11
Line check
14
Airline B
16
Simulator proficiency check
16
Line check
20
Airline C
22
Simulator proficiency check
22
Line check
24
Discussion ............................................................................................................................. 27
Simulator proficiency check results
28
Line check results
29
Limitations
30
Conclusion ............................................................................................................................ 32
References............................................................................................................................. 33
Sources and submissions .................................................................................................. 35
Submissions
35
Australian Transport Safety Bureau .................................................................................. 36
Purpose of safety investigations
36
Developing safety action
36
Glossary................................................................................................................................. 37
ATSB AR-2012-023
Context
Significant debate has occurred within the aviation industry regarding the issues of pilot training
and experience, particularly with regard to the introduction of the Multi-crew Pilot Licence (MPL)
1
by the International Civil Aviation Organisation (ICAO). The main concern being presented by
sectors of the industry that are not in favour of the MPL concept is the possibility that low-hour
pilots are not as competent as their high-hour peers.
As part of this debate, two main issues are currently attracting attention and discussion. The first
is a focus on the hours a pilot has accumulated, generally with an inherent assumption that these
hours ensure a level of individual skill. The second is a focus on competency, both as an assumed
result of flight hour accumulation and because competency-based training is quoted as the
cornerstone of aviation training, in particular with regard to MPL. These issues and assumptions
have received considerable attention following two high profile international aviation accidents in
2009, one involving a Bombardier DHC-8 at Buffalo, New York, and the other an Airbus A330,
2
operating as Air France flight 447 (AF447), en-route from Rio de Janeiro, Brazil to Paris, France.
The AF447 accident highlighted the issue of competency and, in particular, the relevance of
current simulator and non-normal operations training for pilots, as well as raising issues around
automation use and systems knowledge, both of which are subsets of pilot competency. Of
interest is the fact that the flight crew of AF447 had a high number of flight hours, which appears
at odds with the assumption that high flight hours will ensure adequate performance in a nonnormal event.
In response to the DHC-8 accident, which was viewed in part as being the result of an hours/skill
issue, the United States Congress enacted legislation to increase the amount of flying experience
that first officers would need, in order to enter an airline, to 1,500 hours.
3
The Australian Senate held an inquiry into pilot training at the end of 2010 to explore the need to
enact similar legislation in Australia. In response to the Australian inquiry, submissions from flight
training organisations and airlines highlighted the importance of cadets as a recruitment option for
the airlines, a system that has been in place around the world for a number of years. Cadets are
pilots who, generally, have limited or no prior flight training prior to starting their cadetship, and are
trained specifically for an airline at a flight training organisation. After training, these pilots enter
the airline as a second or first officer, depending on the airline and its requirements.
Those supportive of cadet schemes emphasise the ability of the airlines to mould the pilots to the
airline environment, thereby shaping their behaviours and instilling the required performance from
the start of their training. Part of the justification from ICAO in relation to the introduction of MPL
also highlighted these benefits, along with the use of competency-based training as a mitigator of
the risks associated with changes in the training program, and the introduction of multi-crew
cooperation techniques such as crew resource management (CRM) and threat and error
management (TEM) as strong benefits of the changed program.
Those who are not supportive of cadet schemes are of the view that in order to be a successful
airline pilot, an individual needs to have accumulated flight hours either with the military, in general
aviation or other (often smaller) airlines, prior to entering a major airline as a second or first officer.
1
2
ATSB AR-2012-023
Despite the on-going debate and regulatory guidance on pilot training, which outlines the minimum
hours required and the areas that must be covered, there is very little research on the differences,
if any, between pilots in airlines who have entered from these various pathways, including as a
cadet. While there are some studies which have examined the effect of flight hours on accident
risk; judgement; ability to interpret weather and one on the effect of flight hours on performance in
normal flight operations, there is very little looking at performance in non-normal scenarios,
specifically as a function of both flight hours and entry pathway. In this report, entry pathway
refers to the difference between cadets and direct entry pilots. Direct entry pilots enter an airline
having already been licensed and have generally accumulated a number of flight hours. Direct
entry pilots typically build up these hours in the military, general aviation (non-airline flying) or
other airlines.
This report differentiates between normal events, which are routine, expected events that do not
require additional training or specific targeted intervention by a pilot; and non-normal events,
which involve a non-routine event or problem with the aircraft, its systems, other crew, or the
environment, which is not routine and is outside of what is expected in a normal flight.
The aims of this study are to:
(1) test for a difference in performance in the simulator proficiency check for normal and nonnormal operations, based on the entry pathway to the airline
(2) test for a difference in performance in the simulator proficiency check for normal and nonnormal operations, based on the hours experience of the pilots
(3) compare the results with the authors previous study (Todd & Thomas, 2012), in examining if
there are differences in pilot performance in the line checks based on entry pathway or hours.
This study will also address some of the limitations of the previous study, which did not consider
simulator proficiency data and entry pathway.
It can be expected that if the concerns of sectors of the industry relating to low-hour/cadet pilots
are valid, that high-hour/direct entry pilots will perform better. If ICAOs stance on MPL and the
airlines defence of cadets is valid, there will be no difference in performance or cadets will
perform better than their direct entry peers.
ATSB AR-2012-023
Literature review
There have been a number of studies into the effect of experience on performance of pilots,
including the non-technical areas such as decision-making (Wiegmann et al, 2002; Wiggins &
Bollwerk, 2006; Wiggins & OHare, 2003; Todd & Thomas, 2012), with many studying the link
between experience and accident rates (Goh &Wiegmann, 2002; Groff & Price, 2006; Bazargan &
Guzhva, 2011; Li et al, 2003; OHare & Chalmers, 1999). It is important to note that most of these
studies use pilot populations that are quite different to airline pilots. In particular, most of the low
experience groups in the following studies did not receive the same standard of training as airline
cadets who, after initial accelerated competency based training, are transitioned to multi-crew
operations in large, complex aircraft.
One area of interest to researchers in the experience/performance debate is accidents involving
flight into instrument meteorological conditions (IMC) by visual flight rules (VFR) rated pilots. That
is, pilots who are not qualified to fly in instrument conditions, which require them to fly with sole
reference to instruments such as when in cloud, instead of conducting the flight with reference to
visual cues outside the aircraft. The focus of this research has typically centred on the aspects of
decision making and pilot experience, with some studies finding an effect of low experience on
accident risk (Goh & Wiegmann, 2002; Wiegmann et al, 2002).
ATSB AR-2012-023
One study examining both retrospective and prospective indicators of accident risk also used flight
hours as a metric, grouping them into hours flown in the last 6 months, 12 months and a pilots
entire career. The author found that single-engine time in the last 12 months; piston-powered time
for all three hours measures; and landings in the last 6-12 months were all significantly related to
accident involvement (Hunter, 2001). However, an important caveat issued for this study was that
the outcomes may well have been an effect of exposure and in particular a pilot cannot gain the
experience that is usually regarded as a protective factor without this exposure (Hunter, 2001).
A significant effect for expertise was found when differences in perception of weather images as a
function of expertise was examined (Wiggins & OHare, 2002). To differentiate between experts
and novices, they classified experts as pilots who had accumulated greater than 1,000 hours
cross-country flying (mean cross country experience 5,520 hours, mean total experience 6,873
hours) and novices as pilots who had accumulated less than 1,000 hours (mean cross country
experience 281 hours, mean total experience 801 hours).
In a later study examining heuristic-based approaches to in-flight decision making, it was noted
that greater levels of experience tend to be associated with the selection of more optimal
alternatives, irrespective of the information acquisition strategy employed, but that this experience
is defined by the hours accumulated as pilot in command rather than by the number of flight hours
in total (Wiggins & Bollwerk, 2006). The authors also found a significant effect of the hours
accumulated in the 90 days preceding the testing resulted in an increased likelihood the optimal
alternative would be selected, thereby concluding that ...recent experience, rather than total
experience, the license held, or performance during the familiarization scenarios, is most
predictive of the selection of the optimal alternative (Wiggins & Bollwerk, 2006). This is an
interesting finding in relation to the on-going perception within the aviation industry that total flight
hours alone can accurately predict performance and mirrors an earlier study which also noted that
task specific experience was more of a predictor of performance than total experience (Wiggins &
OHare, 1995).
Airline pilots
Interestingly, in a study of air transport pilot involvement in general aviation accidents, one study
found that in general, air transport pilots (ATP) were safer than private pilots (Salvatore et al,
1986). The two groups had comparable exposure, that is, time in general aviation aircraft, but the
ATP group were involved in fewer fatal accidents and fewer pilot-induced accidents, indicating
that the majority of ATP accidents were from factors not related to pilot skill (e.g. aircraft
maintenance). One exception to this appears to be that ATP accidents more often involved
aerobatic flight and for the ATP fatal accidents, nearly 50 per cent occurred during aerobatics. The
authors note that ATPs tended to have less experience in the aircraft type for the fatal aerobatic
accidents, pointing to a consistent finding in terms of experience (Salvatore et al, 1986). The
authors also caution that with an increase in pilot experience ...comes the pilots willingness to
encounter more challenging flight regimes and, perhaps consequently, to accept increased risk
(Salvatore et al, 1986). This note is important to remember during any debate on flight experience
and the performance of more experienced pilots. This concept is highlighted from a different
perspective in another study examining the influence of flight experience on mid-air collision risk
perception, which found that mid-air collisions typically involved highly experienced pilots and that
high-hour pilots were less likely to scan properly for traffic and had become somewhat complacent
given the low perception of risk their experience had provided, and the amount of successful
flights they had completed (Shuch, 1992).
There have been studies in the air transport or airline sector of aviation that examine accident
rates and experience, including one which examined professional pilots risk of an accident as a
function of age and flight experience. This study found that pilots with less than 5,000 hours were
almost twice as likely to be involved in an accident as their more experience colleagues (Li et al,
2003). The authors found a stabilising effect once a total flight time of 10,000 hours had been
accumulated. However, it should be noted that these pilots were all operating in the commuter or
4
ATSB AR-2012-023
air taxi sectors of aviation (that is, not a major airline), which the authors note carries a 10 45
times higher fatal crash rate per pilot flight hour than major airlines.
As part of a gender study examining accident risk for airline pilots, it was found that airline pilots
had less pilot-error accidents as experience (total flying hours) increases. An effect of major
airlines was also found, with pilots who flew for major airlines being less likely to have a pilot-error
accident than those who flew for a non-major airline (McFadden, 1996). Of interest in relation to
airline training, she also commented that once pilots accumulate additional flying hours and
training within the airline business, their pilot-error accident rates decrease markedly (McFadden,
1996). This statement appears to give some support to the idea that cadet training and
development in a wholly airline environment will aid performance, with respect to a lower
involvement in pilot-error accidents. This was also reflected in the findings of Salvatore et al
(1986) above.
A study of pilot performance as a function of flight hours in normal line operations found no
statistically significant differences between low- and high-hour first officers or captains against a
series of technical measures, although there were minor differences in non-technical measures as
a function of crew composition (Todd & Thomas, 2012). There was a difference in the use of
automation by low-hour first officers, who kept the autopilot engaged until a significantly lower
altitude. For this study, low-hour first officers were classified as those with less than 1,500 hours,
and high-hour first officers as those with more than 1,500 hours. For captains, low-hour was
classified as less than 5,000 hours, and high-hour above 5,000 hours (Todd & Thomas, 2012).
Despite the topic of pilot hours and performance attracting significant debate, there is little in the
literature exploring this issue in relation to airline operations.
ATSB AR-2012-023
Methodology
Data sources
Data were collected from three airlines, one of which (Airline A) supplied data for one aircraft type,
and two of which (Airline B and Airline C) supplied data for two aircraft types. All aircraft types are
classified as large passenger carrying jet aircraft capable of conducting high-capacity regular
public transport (RPT) operations.
Performance measures were from simulator checks and line checks from each airline. Deidentified demographic information, which included entry pathway and hours at entry, was also
obtained.
For Airline A, the performance of second officers was measured. For Airlines B and C, the
performance of first officers was measured. All pilots held at least a commercial pilot licence and
none were trained under the multi-crew pilot licence system.
ATSB AR-2012-023
The check and training captains and pilots were blind to the specific hypothesis regarding flight
hours and performance.
Airline C
Data were collected from proficiency check forms for a selection of first officers on one aircraft
type operated by the airline. The data were collected by the operators check and training captains
during the pilots proficiency check in the simulator, which covered non-normal operations, and
during the pilots line check, which covers normal operations.
Inter-rater reliability and calibration was established by the airline and on-going calibration of the
check and training captains by the flight training department occurs yearly. In addition, the check
and training captains are provided with word pictures of performance at each of the five levels of
rating (1=unsatisfactory, 5=excellent) for each measure being tested.
When the data were grouped into entry pathway to test the effect of pathway on performance, a
total of 167 records (cadet n=33, direct entry =134) were completed and considered suitable for
analysis for the simulator sessions (non-normal events) and 251 records (cadet n=56, direct entry
n=195) were considered suitable for the line check data.
When the data were grouped into hours below and above our thresholds to test the effect of hours
on performance, a total of 167 records (low-hour n=26, high-hour n=141) were complete and
considered suitable for analysis for the simulator sessions (non-normal events). For the line check
data, 251 records (low-hour n=51; high-hour n=200) were complete and suitable for analysis.
The check and training captains and pilots were blind to the specific hypothesis regarding flight
hours and performance.
Independent variables
Airline entry pathway
For each airline, to test the effect of training pathway on performance, two groups were created,
one for cadet pilots and one for direct entry.
For Airline A, the measure used to test the effect of pathway on performance was a proficiency
simulator session, which contained a number of metrics listed below under dependent variables,
conducted at the time the pilot entered the airline, and the check to line flight test. Cadet second
officers entered the airline with an average of 246 hours (S.D. 94 h), which they built up during
training, while direct entry second officers had, on average, 3,826 hours (S.D. 1,344 h). Data
about the type of previous flying experience for direct entry pilots into Airline A (airline, general
aviation, military) was not provided, however, the airline reported that the majority of the direct
entry second officers were from a general aviation (or similar) environment. At testing, direct entry
SOs had, on average, 4,729 hours (S.D. 1,427 h) and cadets had, on average, 1,042 hours, (S.D.
431 h).
For Airline B, the measure used to test the effect of pathway was a proficiency simulator session
conducted 5 years after the first officers had joined the airline, and the most recent line check for
these pilots. This time period was chosen to overcome any learning effect as a possible confound.
Cadet first officers entered the airline with, on average 278 hours (S.D. 42 h) predominately built
up during training, while direct entry first officers had, on average, 6734 hours (S.D. 2,853 h). Of
the 81 direct entry first officers, 76 had previous airline experience, 3 were from general aviation
and 2 were from the military. At testing, cadet first officers had, on average, 3,984 hours (S.D. 836
h), and direct entry first officers had, on average, 11,158 hours (S.D. 3,172 h).
For Airline C, the measure used to test the effect of pathway on performance was a proficiency
simulator session, which contained a number of metrics listed below under dependent variables,
conducted during the first five years after entering the airline, and the check to line flight test.
Cadet pilots typically entered the airline with, on average 561 hours (S.D. 457 h) predominately
acquired during training, while direct entry pilots entered with, on average, 5,291 hours (S.D.
ATSB AR-2012-023
3,110 h). Of the 134 direct entry pilots, 32 had previous airline experience, 99 were from general
aviation and 3 were from the military. At testing, cadet pilots had, on average, 1,185 hours (S.D.
511 h) and direct entry pilots had, on average, 6,923 hours (S.D. 3,471 h).
Total flight hours
The effect of flight hours on performance were examined against the same metrics used in the
previous study (Todd & Thomas, 2012). Second officers from Airline A were grouped into those
with below 1,500 hours and those above; and first officers from Airlines B and C into below 1,500
hours and those above. The divide of 1,500 hours was chosen based upon the current legislative
changes in the United States, and the fact that 1,500 hours total time is necessary for pilots to
hold an Air Transport Pilot Licence (ATPL).
Due to the grouping, some high-hour pilots at the time of testing had entered the airline as cadets.
For Airline A, 93 per cent of low-hour second officers had entered the airline as cadets and 3 per
cent of high-hour second officers had entered as cadets. For Airline B, 100 per cent of low-hour
first officers entered as cadets, and 29 per cent of high-hour first officers entered as cadets. For
Airline C proficiency simulator check, 79 per cent low-hour first officers entered as cadets and 5
per cent of high-hour first officers entered as cadets. For the line check data, 91 per cent of lowhour first officers entered as cadets and 2 per cent of high-hour first officers entered as cadets.
Dependent variables
Technical Performance
For each airline, pilots were rated against technical measures during the simulator proficiency
check designed to test technical knowledge and handling, with various normal and non-normal
technical scenarios (Tables 1 to 3). They were assessed using a score range of 1 (unsatisfactory)
to 5 (excellent). In addition, they were also required to demonstrate proficiency in various
regulatory required manoeuvres.
In addition to be above, for Airline B, proficiency was measured by a rating of pass (P), pass with
repeat (1P) or fail (F) for each manoeuvre.
Non-Technical Performance
For each airline, pilots were rated against various non-technical markers and again, pilots were
rated on the 5 point ordinal scale (Tables 1 to 3).
Checks included both normal events, which are routine, expected events that do not require
additional training or specific targeted intervention by a pilot; and non-normal events, which
involve a non-routine event or problem with the aircraft, its systems, other crew, or the
environment, which is not routine and is outside of what is expected in a normal flight (such as an
engine failure).
ATSB AR-2012-023
Table 1:
Communication (N)
Communication (N)
Assertion (N)
Holding (M)
Leadership (N)
Departure (M)
Handover (N)
Teamwork (N)
Stall (M)
Engine failure during cruise (M)
Normal takeoff (M)
Table 2:
Approach (M)
Landing (M)
Holding (M)
Technical (T)
Teamwork (N)
Descent (T)
Configuration (T)
Takeoff (M)
Preparation (T)
Automation (T)
Automation (T)
Handling (T)
Handling (T)
Procedure (T)
Operational (T)
Communication (N)
Command (N)
Airmanship (N)
Briefings (N)
Aircraft (T)
Traffic awareness (T)
Weather assessment (T)
Terrain awareness (T)
Threat awareness (T)
ATSB AR-2012-023
Table 3:
Takeoff (M)
Takeoff (M)
Cruise (T)
Landing (T)
Procedures (T)
Procedures (T)
Communication (N)
Communication (N)
For technical and non-technical performance, the 1-5 ordinal scales were grouped into the
following categories: below standard (1), meeting the standard (2-3), and exceeding the standard
(4-5). These groups were formed on the basis of the rating scales and word pictures used by the
airlines involved in the study.
Statistical analysis
Chi square analyses were utilised for categorical variables and statistical tests were interpreted
using the type 1 error rate of 0.01 for statistical significance. To control for small cell size, Fishers
exact post hoc test was performed on the data. A limitation of using chi square for multiple
comparisons is the increased propensity for type I error to occur, which was why the significance
level of 0.01 was used (Wiggins & Stevens, 1999). However, the results that fall between 0.01 and
0.05 have also been included in the results to highlight overall trends in the data (Mendenhall et
al., 1981).
10
ATSB AR-2012-023
Results
Overall there were no significant differences between the groups for any dependent variable in
relation to meeting the standard required in any of the three airlines. The results below show the
differences in each group exceeding the standard. That is, they all passed, however, performance
varied between the groups within the pass standard. While the line check data from Airline C did
contain some fail items for some pilots, these pilots successfully completed the subsequent check
flight. Significant results are labelled with an asterisk (*) in the figures.
Airline A
Simulator proficiency check
Entry pathway study
Of the seventeen metrics, the performance of cadets differed to the direct entry second officers
with respect to seven of the 17 metrics. The difference was measured as a percentage of pilots
exceeding the standard, with a higher number of direct entry second officers exceeding the
standard compared to cadets (Figure 1). The metrics the groups differed on were engine failure in
the cruise, normal landing, ILS approach, situation awareness, assertion, pilot flying and approach
and landing (Table 4).
Table 4: Differences in performance between cadets and direct entry second officers
Metric
Cadet (%)
2 (1, n=473)
p (2-tailed)
74.4
87.8
9.63
0.004
Normal landing
73.1
88.4
12.50
0.001
ILS approach
69.2
90.6
26.56
<0.001
Situation awareness
75.6
88.4
8.88
0.006
Assertion
78.2
94.7
23.94
<0.001
Pilot flying
79.5
91.6
10.37
0.003
73.1
91.4
21.26
<0.001
For the metric pilot monitoring, the difference was not significant at 0.01, but was within the range
2
of 0.01-0.05 (cadets = 82.1%, direct entry = 90.9%, (1, n = 473) = 5.38 p=0.027, 2-tailed).
11
ATSB AR-2012-023
Figure 1:
12
ATSB AR-2012-023
Low-hour (%)
High-hour (%)
2 (1, n=473)
p (2-tailed)
73.9
87.6
9.00
0.005
71
88.4
14.59
0.001
ILS approach
68.1
90.3
25.93
<0.001
Situation awareness
73.9
88.4
10.39
0.004
Assertion
78.3
94.3
20.54
<0.001
Pilot flying
78.3
91.6
11.27
0.002
73.9
90.8
16.44
<0.001
As above, the metric Pilot monitoring was significant when tested at 0.05 significance (low-hour =
2
81.2%, high-hour = 90.8%, (1, n = 473) = 5.84 p=0.031, 2-tailed).
13
ATSB AR-2012-023
Figure 2:
Line check
Entry pathway study
For the line check data, the only significant difference between cadets and direct entry second
officers was with respect to the one non-technical metric of leadership (cadets = 72.7%, direct
2
entry = 95.4% , (1, n = 197) = 15.26 p=0.002, 2-tailed). The other thirteen metrics showed no
significant difference (Figure 3).
14
ATSB AR-2012-023
Figure 3:
15
ATSB AR-2012-023
Airline B
The data collected and analysed for Airline B was for proficiency and line checks at the 5 year
mark, in order to account for any learning effect that may have been present at airline entry.
16
ATSB AR-2012-023
Figure 5:
17
ATSB AR-2012-023
Figure 6:
Low-hour (%)
High-hour (%)
df (n=169)
p (2-tailed)
Command
10.5
66
27.15
<0.001
CRM
36.8
73.4
12.28
0.003
Situation awareness
26.3
68
22.63
<0.001
System knowledge
63.2
85.3
6.98
0.042
Procedure
10.5
54.6
28.55
<0.001
Handling
15.8
69.3
28.64
<0.001
Automation
63.2
81.3
6.77
0.021
18
ATSB AR-2012-023
Figure 7:
19
ATSB AR-2012-023
Figure 8:
Line check
Entry pathway study
The only significant difference was with respect to the metric of communication, measured by a
higher percentage of cadets exceeding the standard at the 0.05 level (cadets = 95.5%, direct entry
2
= 84.9%, (1, n = 151) = 7.48 p=0.036, 2-tailed) (Figure 9).
20
ATSB AR-2012-023
Figure 9:
21
ATSB AR-2012-023
Airline C
Simulator proficiency check
Entry pathway study
The only significant differences between the cadets and the direct entry groups was that the
cadets scored lower on the measure of leadership and management at the significance level of
2
0.01 (cadet = 44.8%, direct entry = 73.9%, (3, n = 159) = 10.06 p=0.007, 2-tailed). One metric was
2
significant at the 0.05 level only Situation awareness (cadet= 50%, direct entry= 73.9%, (3, n = 166)
= 9.73 p=0.03, 2-tailed). There was no significant difference between the two groups on any of the
other technical or non-technical measures, or for the manoeuvres (Figure 11).
22
ATSB AR-2012-023
23
ATSB AR-2012-023
Line check
Entry pathway study
The only significant differences between the cadets and the direct entry first officers was that the
2
cadets scored lower on the measures of landing (cadet=33.9%, direct entry = 58%, (4, n = 251) =
10.74 p=0.01, 2-tailed) and leadership and management at the significance level of 0.01 (cadet =
2
26.4%, direct entry = 52.6%, (3, n = 169) = 11.17 p=0.003, 2-tailed). There was no significant
difference between the two groups on any of the other technical or non-technical measures, or for
the manoeuvres (Figure 13).
24
ATSB AR-2012-023
25
ATSB AR-2012-023
26
ATSB AR-2012-023
Discussion
The subject of pilot performance is one that often draws heated debate and entrenched positions
amongst the aviation industry. Often these positions have been formed and debates have
occurred without scientific data to support or oppose the various stances. Despite this, the
concept that the performance of low-hour and cadets pilots would be fundamentally different to
their high-hour direct entry colleagues has not previously been explored to any great extent.
While the results of this study indicated some significant differences between the pilot groups for
each airline, it is worth noting that there are a significant number of similarities in their
performance (Table 7 and Table 8). In the line check environment, all three airlines showed very
few differences between the performance of cadets and direct entry pilots, and between low- and
high-hour pilots. This finding is largely consistent with the authors previous study examining the
effect of flight hours on performance in the line operational environment (Todd & Thomas, 2012).
Table 7:
Metric
Airline A
Airline B
Assertion (N)
Table 8:
Airline C
SL
Metric
Airline A
Airline B
Procedure (T)
S
S
Automation (T)
SL
Handling (T)
Airline C
Assertion (N)
27
ATSB AR-2012-023
ATSB AR-2012-023
The overall trend in performance of low-hour first officers may well be due to a similar effect. As
many of the direct entry first officers had previous experience flying in an airline, it is likely they
have had to fly the required manoeuvres a number of times before entry to this airline, as well as
having to demonstrate technical and non-technical proficiency in the simulated environment. It is
worth noting that the low-hour pilots were all cadets (n=19), and 96 of the high-hour pilots (n=150)
had previous airline experience, although the high-hour group also included 43 cadets. It is
therefore possible that the previous airline experience of the high-hour first officers has allowed
these pilots to develop a more in-depth understanding of airline operations and the requirements
of proficiency checks in an airline environment, the inexperience of which may be influencing the
performance of the low-hour pilots.
The difference in performance between the low- and high-hour pilots against all technical and nontechnical metrics is surprising, given there was a lack of significant variation when the effect of
entry pathway was considered on the same dataset. It seems from these results that the effect of
low hours on performance is stronger than any effect of entry pathway, despite all of the low-hour
pilots being cadets.
The difference in performance against the two manoeuvres is quite interesting, given that the
performance of the low-hour pilots was not significantly different against the non-normal
manoeuvres that are generally perceived as being more challenging such as the V1 cut (engine
failure during takeoff) and the one engine inoperative Instrument Landing System Missed
Approach (ILS MAP). This result is also not consistent with the results from the airline entry
pathway study, despite all low-hour pilots being cadets.
For Airline C the only differences in the simulator proficiency checks for technical metrics was at
the 0.05 level of significance and were on the metrics of cruise engine fire (for entry pathway only)
and abnormal emergency (for both groups). Given the number of comparisons and that this result
was at the 0.05 level, it is possible the difference was due to chance only. However, it is also
worth exploring these differences with a view to appropriate training intervention.
ATSB AR-2012-023
hour pilots will be limited in the exposure they have to operating with, and following, SOPs in the
line environment.
For the line check data from Airline A, the difference in performance between the cadets and
direct entry second officers for the metric of leadership could be a function of the role of a second
officer not requiring leadership on the line. Therefore, cadets who have only ever flown as second
officers will not be as familiar with the principles underpinning this metric as pilots who have had
previous exposure to the concept and who have implemented it in previous roles, and/or airline
operations prior to entry into this airline.
That the same difference exists for low-hour pilots from Airline A, with the inclusion of
communication, is not a particularly surprising finding. It is reasonable that low-hour pilots will not
have had great exposure to the line operating environment, unlike many of their high-hour
colleagues and it is probable that this lack of previous exposure is impacting their ability to
communicate effectively. As is the case with leadership, it is possible the influence of role is a
factor here also, specifically that second officers are typically not required to communicate to the
same extent as first officers and captains.
The line check data for Airline C was very similar, with cadets and low-hour pilots scoring lower on
leadership and management and landing metrics. Once again, the prior exposure of direct entry
pilots to the landing phase of flight and to the concept of leadership may be influencing the results.
The only difference in technical metric for line check data was at the 0.05 significance level and
indicated low-hour pilots were scored lower on the visual approach metric. However, given both
groups had a low percentage exceeding the standard (20% low-hour v 38% high-hour), it is
possible there is a difficulty within this metric that is influencing the performance of both groups.
These results are close to those of Airlines A and B, indicating a consistent effect of prior
experience being beneficial.
Overall however, the results from Airline C showed very little difference in performance in both
simulator proficiency checks and line checks for each of the entryway and flight hour groups,
indicating that cadets and low-hour pilots were performing to an acceptable level.
Limitations
It is important that the various figures throughout the results section are interpreted correctly. They
show the percentage of each pilot group for each airline that exceeded the standard. As outlined
previously, the ratings were divided into: below standard (unsatisfactory), meeting the standard
and exceeding the standard. As such, the figures show the percentage of pilots who exceeded the
standard, however, it should be noted that all pilots passed the simulator proficiency or line check
(and therefore at least met the standard) and were suitable to continue flying. It should also be
noted that the idea of a simulator proficiency check is to ensure that a pilot is proficient in a
number of manoeuvres, as well as technical and non-technical skills and knowledge. It is
reasonable to assume there will be an individual variation in performance within this environment
that may well occur due to issues unrelated to individual skill, such as stress or the physical
fitness/wellbeing of the pilot, both of which are beyond the scope of this study to consider.
The other potential influencing factor is that the metrics pilots are rated against are the
overarching areas of proficiency and as such, any sub-element of that metric may influence
performance. For example, if a procedure has recently changed or is not often used, it may be the
execution of the procedure rather than the approach which is causing the pilot to repeat the
manoeuvre, or receive a lower rating. It is therefore possible that the variation in performance that
is being seen is not a function of pilot proficiency or ability, but rather the subtleties of an
underlying procedure and/or other training issues.
Another factor that must be acknowledged is the possibility that one or more of the differences
might be to chance alone, due to the limitation that multiple comparisons were required. However,
30
ATSB AR-2012-023
this was addressed by using the lower significance level of 0.01 with exploration of results that
were significant at 0.05 to ensure overall trends in performance were identified.
The other possible influence on the data, which should be considered, is that of marking bias.
Marking bias by check captains is reported to be an inherent problem in the recurrent measure of
flight crew proficiency checks and generally involves the pilots under check receiving a mark
consistent with their performance for their rank and/or experience. That is, a cadet or low-hour
pilot performing well may only receive a 3, whereas the same performance from a more
experienced first officer or captain may attract a 4. Interestingly the data for Airline B showed a
potential bias of a low-hour stop at 3, with high-hour pilots receiving marks clustered around 3-4.
Therefore the reader should be mindful of the possibility that what the data is showing is a
difference in marking and not in performance.
31
ATSB AR-2012-023
Conclusion
This study has provided an overview of pilot performance as a function of both airline entry
pathway, and flight hours within three high-capacity regular public transport airlines.
There is a level of unease within the wider aviation industry regarding the concept of low-hour
pilots in general, and cadets and MPL pilots in particular. Despite the prevalent existence of cadet
training schemes for a number of airlines around the world for a number of years, there remains a
level of opposition within Australia to such a concept. With the development of training innovations
such as the MPL, this opposition has certainly not abated.
The results indicate that while there are differences in performance between the various groups,
the performance of the cadets and low-hour pilots against their direct entry and more experienced
brethren is remarkably similar. All pilots within the cadet entry and low-hours groups passed the
simulator and line checks required of them, meaning they are proficient to be operating as second
and first officers within the respective airlines, as rated by the airlines check and training staff. The
differences in performance were between meeting the standard and exceeding it, and are subject
to the caveats listed in the discussion. However given that the cadets met the standard and are
therefore proficient, the evidence is demonstrating that the cadet pathway for low-hour pilots is a
valid option for airlines. Contrary to the expressed concerns of some sectors of the industry, there
was no evidence indicating that cadets or low-hour pilots within these airlines were any less
competent or proficient than their direct entry and high-hour peers.
The inclusion of simulator proficiency data and airline entry pathway in this study was to address
some of the limitations from a previous study, which occurred within the line operating
environment only, and without the data to study entry pathway. By studying simulator proficiency
checks, this study was able to examine the performance of cadet and low-hour pilots in nonnormal scenarios, which is not possible during line operations. Aircraft accidents such as AF447
and the DHC-8 accident at Buffalo, New York, highlight the importance of appropriate training for
non-normal events and the need for competent, proficient performance of all flight crew to
successfully handle non-normal events that occur in the operational environment.
The differences in the performance of pilots against all metrics were variable within airlines and
inconsistent across all three airlines. This suggests that the differences between the groups were
not of a systemic nature that would highlight an area of concern for industry. While the metric
normal landing showed a difference across two of the three airlines, none of the rest of the
required regulatory manoeuvres or technical metrics were significantly different in more than one
airline. For non-technical metrics, both leadership and situation awareness were significantly
different in all three airlines. Although this is understandable given the low experience of cadet and
low-hour pilots, focused exposure to those metrics during initial airline training may reduce this
difference as was seen with cadets at the 5-year mark in Airline B.
While differences in performance were discovered as a function of flight hours in some nonnormal scenarios, the fact remains that these differences did not result in the low-hour pilots being
marked as unsatisfactory. Nevertheless, the data from these comparisons may well further inform
future training within high-capacity RPT airline operations to ensure that low-hour pilots are
receiving suitable training interventions for the areas they are not performing as well on.
While the subject of low-hour and cadet pilots have received significant international attention
recently (e.g. the Australian senate inquiry and American legislative changes), it is important to
ensure that any on-going debate is informed by relevant studies and science. With the
advancement of aircraft technology and a growing shortage of pilots; regulators, airlines and
industry bodies need to ensure that the underpinning training aids pilots by providing targeted
training to prepare them for any non-normal events during flight.
32
ATSB AR-2012-023
References
Air Line Pilots Association. (2009) ALPA White Paper: Producing a Professional Airline Pilot.
Washington DC, US: Air line Pilots Association, International.
Bazargan, M., & Guzhva, V.S. (2011). Impact of gender, age and experience of pilots on general
aviation accidents. Accident Analysis and Prevention, 43:962-970.
st
BEA (2012). Final Report on accident on 1 June 2009 to the Airbus A330-203 registered FGZCP operated by Air France flight AF447 Rio de Janeiro Paris. Bureau dEnqutes et
dAnalyses pour la scurit de laviation civile. Paris, France.
Causse, M., Dehais, F. & Pastor, J. (2010). Flight experience and executive functions predict flight
simulator performance in general aviation pilots. In: 4th International Conference on Research in
Air Transportation ICRAT 2010, 04 June 2010, Budapest, Hungary.
Clarke, D.D., Ward, P., Bartle, C. & Truman, W. (2006). Young driver accidents in the UK: The
influence of age, experience, and time of day. Accident Analysis and Prevention; 38:871-8.
Dinges, T.A., Hulse, M.C., Mollenhauer, M.A., Fleishman, R.N., McGehee, D.V. & Manakkal, N.
(1997). Effects of Age, System Experience, and Navigation Technique on Driving with an
Advanced Traveler Information System. Human Factors 39(2):177-99.
Dismukes, R.K., Berman, B.J. & Loukopoulos, L.D. (2007). The Limits of Expertise: Rethinking
pilot error and the causes of airline accidents. Ashgate, Aldershot, UK.
Fisher, D.L., Laurie, N.E., Glaser, R., Connerney, K., Pollatsek, A., Duffy, S.A. & Brock, J. (2002).
Use of a Fixed-Based Driving Simulator to Evaluate the Effects of Experience and PC-Based Risk
Awareness Training on Drivers Decisions. Human Factors, 44(2): 287-302.
Goh, J. & Wiegmann, D. (2002). Human Factors Analysis of Accidents Involving Visual Flight
Rules Flight into Adverse Weather. Aviation, Space, and Environmental Medicine, 73:817-22.
Groff, L.S. & Price, J.M. (2006). General Aviation Accidents in Degraded Visibility: A Case Control
Study of 72 Accidents. Aviation, Space, Environmental Medicine, 77:1062-7.
Hunter, D.R. (2001). Retrospective and Prospective Validity of Aircraft Accident Risk Indicators.
Human Factors, 43(4), 509-518.
th
ICAO (2006). Annex 1 Personnel Licensing (10 edition). International Civil Aviation
Organization, Montreal, Canada
Kennedy, Q., Taylor, J.L, Reade, G. & Yesavage, J.A. (2010). Age and Expertise Effects in
Aviation Decision Making and Flight Control in a Flight Simulator. Aviation, Space, and
Environmental Medicine, 81(5), 489-497.
Li, G., Baker, S.P., Grabowski, J.G., Qiang, Y., McCarthy, M.L. & Rebok, G.W. (2003). Age, Flight
Experience, and Risk of Crash Involvement in a Cohort of Professional Pilots. American Journal of
Epidemiology, 157:874-880.
McFadden, K.L. (1996). Comparing Pilot-error Accident Rates of Male and Female Airline Pilots.
Omega, International Journal of Management Science, 24(4), 443-450.
Mendenhall, W., Scheaffer, R.L. & Wackerly, D.D (1981). Mathematical Statistics with Applications
(second edition). PWS Publishers, Boston, United States.
NTSB (2012). Loss of Control on Approach Colgan Air, Inc. Operating as Continental Connection
Flight 3407, Bombardier DHC-8-400, N200WQ, Clarence Center, New York, February 12, 2009.
National Transportation Safety Board, Washington, United States of America.
33
ATSB AR-2012-023
OHare, D. & Chalmers, D. (1999). The Incidence of Incidents: A Nationwide Study of Flight
Experience and Exposure to Accidents and Incidents. International Journal of Aviation
Psychology, 9:1-18
Parliament of Australia Senate. (2010). Pilot training and airline safety. Retrieved 20 October 2010
from http://www.aph.gov.au/senate/committee/rat_ctte/pilots_2010/info.htm
Salvatore, S., Stearns, M.D., Huntley Jr, M.S. & Mengert, P. (1986). Air transport pilot involvement
in general aviation accidents. Ergonomics, 29(11), 1455-1467.
Shuch, H.P. (1992). The Influence of Flight Experience on Midair Collision Risk Perception.
Accident Analysis and Prevention, 24(6), 655-660.
Sutton O. (2005). Shortcut or Fast Track? Air Transport World, 2005. Retrieved 24 April 2008 from
http://www.atwonline.com/channels/safetySecurity/article.html?articleID=1428
Todd, M.A. & Thomas, M.J.W. (2012). Flight Hours and Flight Crew Performance in Commercial
Aviation. Aviation, Space, and Environmental Medicine, 83(8), 776-782.
United States of America Congress. (2010). Airline Safety and Federal Aviation Administration
Extension Act of 2010. Washington, DC: United States Government Printing Office.
Wiegmann, D.A., Goh, J. & OHare, D. (2002). The Role of Situation Assessment and Flight
Experience in Pilots Decisions to Continue Visual Flight Rules Flight into Adverse Weather.
Human Factors, 44(2), 189-97.
Wiggins, M.W. & Bollwerk, S. (2006). Heuristic-Based Information Acquisition and Decision
Making Among Pilots. Human Factors, 48(4), 734-46.
Wiggins, M.W. & OHare, D. (2003). Expert and Novice Pilot Perceptions of Static In-Flight Images
of Weather. International Journal of Aviation Psychology, 13:173-87.
Wiggins, M.W. & Stevens, C. (1999). Aviation Social Science: Research Methods in Practice.
Ashgate, Aldershot, England.
34
ATSB AR-2012-023
35
ATSB AR-2012-023
36
ATSB AR-2012-023
Glossary
AF447
ATP
ATPL
CRM
GA
General Aviation
ICAO
ILS
ILS MAP
IMC
MPL
RPT
S.D.
Standard Deviation
SOP
TEM
VFR
37
Research
AR-2012-023
Final 17 July 2013