SIDs 2020 Paper 64red
SIDs 2020 Paper 64red
SIDs 2020 Paper 64red
Figure 2. The proposed framework for monitoring aircraft turnaround process. By detecting aircraft, ground support equipment and recognizing aircraft type,
the framework can detect activities and predict push-back time.
Figure 4. Videos collected from a live camera in Tokachi-Obihiro airport with Figure 5. Objects are labelled as bounding boxes. The bounding boxes are
different aircraft types, time and weather conditions. heavily overlap which makes both labelling and detecting difficult.
Figure 7. The AirNet recognizer adopted from [12] for aircraft type recognition.
V. R ESULTS
A. Aircraft Recognition and Object Detection
As mentioned in Section IV, we use the lightest version
Figure 12. Detected activities is displayed by the framework starting when (i.e., a minimal number of trainable parameters) of AirNet for
an aircraft arrives (second 0) and ending when an aircraft pushes back (green
star) over time (from blue to red).
the aircraft type recognizer. At this minimal configuration, the
recognizer yields an accuracy of 100% in detecting two aircraft
types. If there are more than two aircraft types, high detection
in/out signifies refueling. We do not distinguish between aft accuracy is still achievable by using a more heavyweight
or forward cargo; therefore, we display those as the first cargo version of AirNet, i.e., introducing more trainable parameters.
and the second cargo. Activities that have attachment and Figure 14 shows the precision-recall curves of the detection
detachment are merged as a duration. for different objects. A high recall detector aims to detect
as many objects as possible, which helps to reduce false
C. Push-back Prediction negative. In contrast, a high precision detector aims to detect
From the TA history and manual, we establish the rela- objects as precise as possible, which helps to reduce false
tionship between the time required to finish activities and the positive. A precision-recall curve indicates the trade-off be-
push-back time. Based on the TA history time, we calculate tween precision and recall, which is important in determining
the average time to complete each activity and the ratios of the detection threshold. In Figure 14, the top-right corner
these average times to the actual push-back time, as shown in is the most desirable region, where the detector is able to
Table I. Additionally, the predicted push-back time could not achieve high precision and high recall simultaneously. Since
be shorter than the manual push-back time from APM. different thresholds are associated with different precision and
The predicted time is updated in real-time, as shown in recall values, Average Precision is an important metric that
Figure 13. First, it is initialized to the departure flight schedule calculates the average precision over all possible thresholds. In
on the airport website. It remains at this value until the first other words, Average Precision, ranging from 0 to 1, indicates
activity (refueling) finishes. Since the refueling activity ends the detector’s performance regardless of detection thresholds
early, the predicted push-back time drops significantly. After (i.e., the higher Average Precision the better performance).
Table II shows the results of object detection, achieving a high-
performance level with a mean Average Precision of 0.9514.
TABLE I. R ATIO BETWEEN FINISH TIME OF ACTIVITIES AND ACTUAL
PUSH - BACK TIME . Large objects (such as aircraft, bridge, tow truck, and fuel
truck) are detected with very high precision. Cargo loader and
Activities Bridge 1st Cargo Loader 2nd Cargo Loader Refueling
Ratio 0.9263 0.5550 0.8319 0.3392 cargo truck are detected with lower precision than other large
objects as they suffer from intra-variance and overlapping. Due
7
TABLE II. AVERAGE PRECISION OF AIRCRAFT AND DIFFERENT GROUND SUPPORT EQUIPMENT.
Class Aircraft Bridge Cargo Loader Cargo Truck Fuel Pipe Fuel Truck Tow Bar Tow Truck Mean
Average Precision 0.9996 0.9998 0.9304 0.9143 0.9028 0.9989 0.8663 0.9988 0.9514
Figure 15. The errors between detected and ground truth activities. Negative
errors indicate activities are detected earlier than ground truth, and vice versa.
Figure 14. Precision-recall curves of the detection for different objects. The
area under each curve indicates the performance of the corresponding detector
(the larger area the better accuracy). seconds compared to 2280 from the manual APM. Inter-
estingly, the difference between actual and scheduled push-
back times seems to be variable, regardless of duration. For
to their small sizes and heavy overlapping by larger objects, example, in the two figures with actual times of approximately
the fuel pipe and tow bar have the lowest detection precision. 2700 seconds, the scheduled times range from 2917 to 3506
However, this does not affect the activity detection as we use seconds; while in the two figures with schedule times of
different methods to detect refueling, and the tow bar does not about 3600 seconds, the actual times range from 2686 to 3446
contribute to the activity detection. seconds.
Moreover, there is no standard process for aircraft
B. Activity Detection turnaround. For example, in the upper right chart of Figure 16,
As ground truth activity time is labeled as a duration, while refueling time and first cargo loader time are similar to each
the detected activity time is estimated instantly, we normalize other, while in the lower-left chart of Figure 16, the bridge
the ground truth activity time using the middle point of that detaches before the second cargo loader does. The framework
duration. We divide nine activities into four groups, namely can get more accuracy in predicting push-back, compared
bridge (bridge attachment/detachment), cargo loader (cargo to the scheduled time, from the time of the first activity
loader attachment/detachment), refueling (drawing in/out of completion (refueling) in two of the five videos. Among these
fuel pipe), and other (aircraft arrival, tow truck connection, two videos, it can more accurately predict push-back from
and push-back). the time the second cargo loader detaches in one video and
from the bridge detaches in the remaining video compared to
Figure 15 shows the errors between detected and ground
the scheduled time. It results in a more accurate push-back
truth activities represented as box plots. The errors are cal-
prediction being available in four out of the five videos.
culated by subtracting the predicted activity time from the
ground-truth, as we want to keep both the magnitude and sign. VI. C ONCLUSION
Therefore, negative errors indicate activities that are detected We have proposed a novel computer vision-based frame-
earlier than the ground truth, and positive errors indicate work that can monitor the aircraft turnaround process, includ-
activities that are detected later. As can be seen, the median ing object detection, activity detection, and push-back predic-
errors for the Bridge, Cargo Loader, and Other groups range tion. By using ConvNets, aircraft type recognition achieves
from 1.5 to 2 seconds. As the fuel pipe has low detection 100% accuracy, while object detection achieves 0.9514 mean
performance, refueling errors are not stable, resulting in more Average Precision. For activity detection, the median error is
outliers. It results in a median error of the refueling time, only smaller than 6 seconds, which can be considered very low in
5.5 seconds, which is still accurate compared to the overall the overall turnaround duration. Furthermore, the framework
turnaround duration. provided more accurate push-back prediction that the airport
schedule in four out of five, or 80% of cases.
C. Push-back Prediction
In the future, we intend to further improve push-back
As there are only five testing videos, we display all these prediction time by collecting Gate CAM videos from differ-
videos’ prediction results in Figure 13 and Figure 16. Based on ent airports and incorporating information from the A-CDM
the five videos, one can see that the departure flight schedule system such as Target Start-up Approval time and runway
times are very different, ranging from 2855 seconds to 3898 configuration and availability. It may lead to a reduction in
8
Figure 16. Push-back time prediction of different TA processes. The difference between actual and schedule push-back times seems to be variable, regardless
of duration.
the aircraft waiting time at the runway holding points, reduce [10] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for
fuel consumption on the taxiways, and improve the passenger convolutional neural networks,” arXiv preprint arXiv:1905.11946, 2019.
[11] M. Tan et al., “Efficientdet: Scalable and efficient object detection,”
experience by having a smoother departure flow. in Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, 2020, pp. 10 781–10 790.
ACKNOWLEDGMENT [12] P. Thai et al., “Deep4air: A novel deep learning framework for airport
airside surveillance,” 2020.
This work was conducted under the Saab-NTU Joint Lab [13] Y. LeCun et al., “Gradient-based learning applied to document recogni-
with support from Saab AB, Saab Singapore Pte Ltd., and Air tion,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
Traffic Management Research Institute, Nanyang Technologi- [14] J. Deng et al., “Imagenet: A large-scale hierarchical image database,”
in IEEE conference on computer vision and pattern recognition, 2009.
cal University, Singapore. [15] A. Krizhevsky et al., “ImageNet classification with deep convolutional
neural networks,” Communications of the ACM, vol. 60, no. 6, pp.
R EFERENCES 84–90, May 2017. [Online]. Available: http://dl.acm.org/citation.cfm?
doid=3098997.3065386
[1] M. Schmidt, “A review of aircraft turnaround operations and simula- [16] C. Szegedy et al., “Going Deeper with Convolutions,” arXiv:1409.4842
tions,” Progress in Aerospace Sciences, vol. 92, pp. 25–38, 2017. [cs], Sep. 2014, arXiv: 1409.4842.
[2] Boeing. (2011) Airplane characteristics for airport planning. https: [17] K. He et al., “Deep Residual Learning for Image Recognition,”
//www.boeing.com/commercial/airports/plan manuals.page. arXiv:1512.03385 [cs], Dec. 2015, arXiv: 1512.03385. [Online].
[3] M. M. Mota et al., “Simulation-based turnaround evaluation for lelystad Available: http://arxiv.org/abs/1512.03385
airport,” Journal of Air Transport Management, vol. 64, pp. 21–32, 2017. [18] A. G. Howard et al., “Mobilenets: Efficient convolutional neural net-
[4] S. Okwir et al., “Managing turnaround performance through collabora- works for mobile vision applications,” arXiv preprint:1704.04861, 2017.
tive decision making,” Journal of Air Transport Management, vol. 58, [19] F. Chollet, “Xception: Deep learning with depthwise separable convolu-
pp. 183–196, 2017. tions,” in Proceedings of the IEEE conference on computer vision and
[5] M. Schultz and S. Reitmann, “Machine learning approach to predict pattern recognition, 2017, pp. 1251–1258.
aircraft boarding,” Transportation Research Part C: Emerging Technolo- [20] S. Ren et al., “Faster r-cnn: Towards real-time object detection with
gies, vol. 98, pp. 391–408, 2019. region proposal networks,” in Advances in neural information processing
[6] B. Oreschko et al., “Turnaround prediction with stochastic process systems, 2015, pp. 91–99.
times and airport specific delay pattern,” in International Conference [21] T.-Y. Lin et al., “Feature pyramid networks for object detection,” in
on Research in Airport Transportation (ICRAT), Berkeley, 2012. Proceedings of the IEEE conference on computer vision and pattern
[7] A. Koutsia et al., “Automated visual traffic monitoring and surveillance recognition, 2017, pp. 2117–2125.
through a network of distributed units,” in ISPRS. Citeseer, 2008. [22] ——, “Microsoft coco: Common objects in context,” in European
[8] N. Pavlidou et al., “Using Intelligent Digital Cameras to Monitor conference on computer vision. Springer, 2014.
Aerodrome Surface Traffic,” IEEE Intelligent Systems, vol. 20, pp. [23] P. F. Felzenszwalb et al., “Object detection with discriminatively trained
76–81, 2005. [Online]. Available: http://ieeexplore.ieee.org/document/ part-based models,” IEEE transactions on pattern analysis and machine
1439483/ intelligence, vol. 32, no. 9, pp. 1627–1645, 2009.
[9] H.-L. Lu et al., “Airport gate operation monitoring using computer [24] P. Sermanet et al., “Overfeat: Integrated recognition, localization and de-
vision techniques,” in 16th AIAA Aviation Technology, Integration, and tection using convolutional networks,” arXiv preprint:1312.6229, 2013.
Operations Conference, 2016, p. 3912.