Feature Article: DOI. No. 10.1109/MAES.2020.

3008468

Towards the Use of Artificial Intelligence on the Edge

in Space Systems: Challenges and Opportunities
Gianluca Furano, European Space Agency
Gabriele Meoni, University of Pisa
Aubrey Dunne, Ubotica Technologies Ltd
David Moloney, Intel Ireland Ltd.
Veronique Ferlet-Cavrois, European Space Agency
Antonis Tavoularis, European Space Agency
Jonathan Byrne, Intel Ireland Ltd.
Le
onie Buckley, Ubotica Technologies Ltd.
Mihalis Psarakis, University of Piraeus
Kay-Obbe Voss, GSI Helmholtz Centre
Luca Fanucci, University of Pisa

markets, thanks to the decreased development time and

INTRODUCTION
launch costs [1], [2]. In this scenario, one of the most
Technological improvement is an enabler for the “New significant limiting factors for the remote sensing mar-
Space” era [1]. Modern silicon allows for embedded platforms ket, especially for miniaturized spacecraft, is represented
featuring very large computing power and reduced dimension, by the up- and downlink bandwidth and onboard com-
power consumption, and costs. The improved computing pute capability on satellites combined with ground sta-
capabilities have increased the scope of possible applications tion availability [3], [4]. For Cubesats, additional
Earth observation (EO), while the miniaturization of process- limitations are due to the reduced system power budget,
ing platforms has permitted to shrink the size and mass of which directly limits the maximum downlink through-
spacecraft, enabling the production of small but capable satel- put [4]. This article investigates the use of artificial intel-
lites, in mass classes well below the 10 Kg (SmallSats) [1], ligence (AI) with particular focus on deep neural
[2]. SmallSats are now affordable, even from private compa- networks (DNNs) on board spacecraft, analyzing the
nies, causing the appearance of new actors in the space possible benefits in terms of bandwidth downlink due to
the on-the-edge processing of data [5]: a novel AI-based
architecture for EO satellites is proposed, which embeds
Authors’ current addresses: Gianluca Furano, Veronique AI algorithms for consuming data at source rather than
Ferlet-Cavrois, Antonis Tavoularis, Data Systems Divi- on the ground, allowing the deployment of value-added
sion of European Space Research and Technology Cen- applications in space which utilize a tiny fraction of the
tre, European Space Agency, 2201 AZ, Noordwijk, The downlink bandwidth that would be otherwise required.
Netherlands (e-mail: [email protected]). Jonathan The proposed solution has the potential to revolutionize
Byrne, David Moloney, Intel Ireland Ltd., Kildare W23 EO and remote sensing, reducing the time and cost to
CX68, Ireland. Kay-Obbe Voss, GSI Helmholtz Centre, deploy new applications to space by orders of magnitude
64291 Darmstadt, Germany. Aubrey Dunne, L
eonie
compared with the current state-of-the-art. This article
Buckley, Ubotica Technologies Ltd., Dublin D11 KXN4,
also presents the Myriad 2 case study, a commercial off-
Ireland. Mihalis Psarakis, Department of Informatics,
the-shelf (COTS) vision processing unit (VPU) used for
University of Piraeus, 18534 Piraeus, Greece. Gabri-
ele Meoni, Luca Fanucci, Information Engineering the acceleration of convolutional neural networks
Department, University of Pisa, 56122 Pisa, Italy. (CNNs). It finally reports the first results in radiation tol-
Manuscript received August 27, 2019, revised February erance and power/performance of these COTS devices
10, 2020, June 15, 2020, and ready for publication July for space-based applications, showing how, despite
9, 2020. their non-space nature, such devices can be effectively
Review handled by Dietrich Fraenken. used for low Earth orbit (LEO) non-mission-critical
0885-8985/20/$26.00 ß 2020 IEEE applications.

44 IEEE A&E SYSTEMS MAGAZINE DECEMBER 2020

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Credit: Image licensed by Ingram Publishing

sufficiently mitigate SEEs risks, the architecture of space

SPACE-BASED COMPUTE PLATFORMS
computing platforms often features hardware and informa-
tion redundancy techniques, such as triple modular redun-
STATE OF THE ART IN SPACE ELECTRONICS dancy and error correction codes, which increase the
Space-qualified devices shall be designed to be tolerant to complexity of the design, at expenses of performance [14].
deep vacuum conditions, strong vibrations, and higher tem- This is exacerbated by the use of larger technology nodes
perature ranges, to mitigate electrostatic discharges risks, and of RHBD cells, which results, for example, in smaller
and to guarantee sufficient heat dissipation exclusively cache memories of rad-hard processors. Finally, processors
through thermal conduction [6], [7]. Furthermore, modern used in space are typically slower due to the long qualifica-
silicon in space faces specific challenges due to ionizing tion process for space-grade components and to the risk-
radiation: space-borne devices shall take into account both averse design choices of the space industry, preferring com-
single event effects (SEEs) caused by the charge deposited ponents and technologies, which already have a proven
by ionizing particles [8], and effects due to long time expo- space history. Thus, the space processors typically lag more
sure to radiation, such as total ionizing dose (TID) and total- than a decade behind their commercial counterparts in terms
not ionizing dose [9], [10]. SEE include temporary effects, of performance and the gap is widening every year (e.g., the
such as single event transients (SETs), single event upsets commercial PowerPC-750 roadmap showed a performance
(SEUs), and single event functional interrupts (SEFIs) but improvement of
2400 from the early 1990s to 2005 [15],
also not reversible effects, such as single event latchups whereas the rad-hard version was improved by only
(SELs), which might be destructive for devices [8]. SETs
300 [6]). An analysis of space-qualified computing plat-
are logical transients due to ionizing radiation. SEUs are bit- forms follows.
flips in registers and memories that are caused either when
the transient pulses are captured by sequential circuits or
because the charge of the storage elements is discharged by
SPACE-QUALIFIED PROCESSORS
ionizing radiation strikes. When such errors affect control Current space-grade data processing systems use quali-
logic, causing changes in the state of the system, SEUs are fied parts such as GR740, which represents the ESA
generally called SEFIs. Space-qualified devices should then Next Generation MicroProcessor shown in [16]. The
apply Radiation-Hardened By Design (RHBD) [11] or development of GR470, which was started in 2009
Radiation-Hardened By Process (RHBP) techniques, for with the first prototypes delivered in 2013, was a tran-
example, by exploiting flash-based and silicon on insula- sition from single-core space microprocessors to multi-
tor [12] processes. The high specificity of such techniques core, high-performance processors. GR740 is now an
and the low addressable market scopes lead to significantly application-specific integrated circuit (ASIC) featuring
higher costs than standard complementary metal-oxide a quad-core LEON4 processor and other communica-
semicondutor (CMOS) designs [13]. Moreover, to tion interfaces. The use of GR740 is planned for future

DECEMBER 2020 IEEE A&E SYSTEMS MAGAZINE 45

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Towards the Use of Artificial Intelligence on the Edge in Space Systems: Challenges and Opportunities

missions, such as those announced by ArianeGroup and Mpixels/s/W/kg/$ metric. To understand the relevance of
PTScientists [17] and ROSKOSMOS [18]. Current this metric, it is possible to consider that
spacecraft architectures are mostly based on the Mpixels/s
 W indicates the computing efficiency of the
LEON3FT processor [10]. Both LEON3 and LEON4
platform. Benchmarking the performance/Watt is
processors are based on the open standard SPARC V8
necessary in view of the importance of this factor
Reduced Instruction Set Computer (RISC) Instruction
for SmallSats [4].
Set Architecture (ISA), whose choice was due to its
maturity, simplicity, and by the possibility to avoid  Computing platform mass (kg) reduction is a must,
cross-compiling in view of the availability of SPARC- due to its direct impact on launch costs and minia-
based workstation [10]. However, SPARC ISA did not turization possibilities [2].
find a broad use outside space fields, differently from
 Costs ($) reduction has great importance for the
what it was supposed in the late 1990s [10]. For this
advent of the New Space era [1], [13]. Payload com-
reason, products such as NOEL-V, which is based on
mand and data processing platforms can influence
RISC-V ISA, were conceived, fact that arises a broad
the total cost of the satellite by more than 12% [10].
interest in several industrial applications.
The computing platform of upcoming ESA lunar rego-
SPACE-QUALIFIED FPGA lith miner missions contains a computer vision subsystem
that weighs
35 kg, dissipates
60 W, can process less
Space data processing systems usually exploit field pro- than 4 frame per second (FPS) at a frame resolution of 1
grammable gate arrays (FPGAs) as coprocessors for the Mpixel, and has a processing hardware cost estimated at in
execution of several computationally intensive algorithms, excess of
$200 000 [22], [23]. This results in a value for
which require to be executed by specific hardware archi- the above performance metric of 9.52e9 for this vision
tectures. Examples of such algorithms include high-speed subsystem ((4 Mpixel/s)/(35 kg)/(60W)/($200 000)). In
communication and file delivery protocols [19], data com- contrast, the Myriad 2 System On a Chip (SoC), which is
pression, and signal processing applications [20]. As pre- designed from the ground up for processing efficiency per
viously introduced, compared to COTS space-qualified watt per dollar, would have a corresponding performance
FPGAs exploit larger technological nodes and specific metric of 250 000 ((600 Mpixels/s)/(0.0002 kg)/(1.2 W)/
processes and designs to guarantee sufficient tolerance to ($10)). This massive disparity demonstrates the appeal of a
radiations. In particular, different approaches exist. Micro- lightweight, low-cost, low-power COTS solution for many
semi FPGAs exploit a flash-based design to increase TID non-mission-critical space applications from launch to
tolerance and a specific CMOS design to guarantee high orbit and potentially beyond [24]. ESA has successfully
SEEs resistance [11]. The last Microsemi releases are the used commercial parts in spacecraft for specific and some-
PolarFire FPGA and RTG4 realized in 28 nm and 65 nm times mission-critical applications throughout the
processes, respectively. NanoXplore exploits an RHBD Agency‘s history. Although the number of COTS devices
static random access memory (SRAM)-based technology flown on ESA missions is considerable, this has been
process to provide higher performance. NanoExplore achieved by careful selection, qualification, and screening,
products involve NG-MEDIUM and NG-LARGE FPGA, and few key COTS parts have been used in largely non-
respectively, in 65 nm and 28 nm processes. COTS designs. The level of screening is quite well charac-
terized in existing ESA standard parts documents such as
ECSS-Q-ST-60-13 C. Despite the potential advantages of
COTS ELECTRONICS FOR SPACE APPLICATIONS COTS processing systems over space-grade devices, to
The market pressure on deep learning (DL) in commercial extend the applicability of DL to missions having more
applications led many electronic devices producers to stringent requirements in terms of radiation resistance,
invest in mostly application-specific hardware accelerators some approaches to deploy DNNs on space-qualified devi-
to bring efficient inference of DNNs at the edge. Because ces were conceived. For instance, Blacker et al. [25] devel-
of the high market scope, such devices offer remarkable oped a tool to facilitate the porting of DNNs onboard the
performance/power consumption tradeoffs at a reduced LEON 3 processor.
price, which make them able to outperform standard
graphics processing units (GPUs) and central processing
units (CPUs) both for on-the-edge and cloud processing of
machine learning (ML) algorithms [21]. To make evident
STATE OF THE ART OF COTS SOCS FOR DNN INFERENCE
the potential of such devices for space applications, it is The COTS hardware solutions for DNN inference on the
possible to analyze the equipment composed by space- edge can be classified based on their core computing
only hardware for the already cited ESA lunar regolith unit [26]: 1) ASIC-based hardware that integrates custom
miner mission mooted for 2025 [17],[18] by using a AI chips; 2) GPU-based hardware that uses the massive

46 IEEE A&E SYSTEMS MAGAZINE DECEMBER 2020

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Furano et al.

computing parallelism of the GPUs to improve perfor- floating point, and 8-bit fixed point supports. Myriad X can
mance at the cost of higher energy consumption; and 3) reach up to 4 tera operation per second (TOPS), with a
FPGA-based hardware that significantly increases the per- maximum power consumption of 1.5 W. While the Myriad
formance per watt but with limited programming capabil- 2 implements the CNN pipeline largely in software on the
ity compared to GPUs. SHAVE processors, the Myriad X device is the first device
to bring the CNN inference hardware to market offering
ASIC-BASED AI PLATFORMS aggregate 8 more performance for CNN inference within
the same power envelope and form-factor. Intel’s NCS2 is
According to [27], a large number of companies have been available to exploit Myriad X for embedded applications.
involved in the development of silicon AI accelerators.
These accelerators are in the form of IP cores for integration
in SoCs, single chips, or board-level platforms and integrate GOOGLE CORAL
an AI engine, which can compute fast vectors and matrices Google Coral is a 28-nm Tensor Product Unit (TPU) opti-
operations. These accelerators target mobile devices (e.g.,
mized for the inference of DNNs. Differently from Intel’s
Qualcomm’s Snapdragon 855, HiSilicon’s Kirin 970, and Myriad, Google Coral is based on systolic arrays and fea-
MediaTek’s Helio P90), IoT edge applications (e.g., AI tures around 8 MB on board memory. Google Coral dem-
AM5729 of BeagleBone) or the general use case (e.g., onstrated performances up to 4 TOPS, featuring a power
Intel’s Myriad 2 and Google’s Coral) [28]. In this article, consumption of 2 W [29].
we focus on the latter case as most appropriate for use on
the edge in space systems and illustrate the Intel Myriad
platforms, as our demonstration vehicle for the radiation GPU-BASED PLATFORMS
experiments and the Google Coral platform as a second rep-
NVIDIA Jetson is the most commonly used GPU-based
resentative example. The reader can refer to [28]–[30] for a
AI accelerator. It is a 20-nm Maxwell GPU, containing
more detailed description and comparison of the AI chips.
128 CUDA processors. It is equipped with a Quad-core
ARM Cortex-A57 MPCore, which shares with the GPU
MYRIAD 2. Myriad 2 is a system-in-package (SiP) device 4 GB LPDDR4 memory. NVIDIA Jetson can provide 472
that contains a 28-nm SoC die fabricated in a standard GFLoating Point OPerations (FLOPs), dissipating up to
LPCMOS TSMC manufacturing process along with a 5 W [29].
stacked 512 MB Low Power Double Data Rate (LPDDR)3
dynamic random access memory (DRAM) die [31].
The Myriad 2 architecture contains two LEON4 processors, COTS FPGAs
one for user-applications running under RTEMS real-time The use of COTS FPGAs to accelerate DNNs was investi-
operating system, and the other running a scheduler for the gated in the literature [32]–[35]. Compared to space-quali-
image signal processing (ISP) and computer vision pipe- fied FPGAs, COTS FPGAs exploit more modern
line. In addition to the LEON4s, the device contains 12 technology processes, offering improved computational
streaming hybrid architecture vector engine (SHAVE) very power and better power consumption/performance trade-
long instruction word (VLIW) digital signal processor offs and higher resource budgets at lower prices. Different
(DSP) processors capable of executing the equivalent of approaches are possible to deploy DNNs on board FPGAs.
144 RISC instructions in a single 600-MHz clock-cycle. For small-footprint models, fully on-chip designs, exploit-
Finally, an ISP pipeline can process an aggregate 1 pixel ing on-chip memories only, are possible [34]. The work
per cycle at the 600-MHz system clock rate for presented in [32] compare the FPGA approach to a solution
600 Mpixel/s aggregate throughput. The ISP hardware con- based on the Intel NCS1 for small-size models, demonstrat-
tains streaming image processing pipeline (SIPP) debayer- ing the benefits of FPGAs in terms of inference time and
ing, polyphase up/down-caling filters, convolution filters, energy-efficiency. For more complex DNNs, solutions
etc. Intel’s Neural Compute Stick (NCS)1 enables the relying on external memories to store coefficients and inter-
exploitation of Myriad 2 for neural network inference in a mediate results are generally exploited [33]. Generally, the
universal serial bus (USB) form factor. advantages of FPGAs compared to other COTS strictly
depend on many factors, such as the architecture, model
MYRIAD X. Myriad X represents the next generation of size, and the number of bits used for representation [30],
Intel hardware accelerators for DNN inference. The [32], [34], [35]. In view of this large design space, the use
smaller the technology node (16 nm), the higher number of of FPGAs usually requires a much longer development
SHAVE processors and the availability of a novel 512 MB time compared to Myriad 2 and other COTS devices, when
LPDDR4 memory provide Myriad X an increased effi- an approach based on a design from scratch is
ciency and computational power with respect to a Myriad exploited [32], [34]. Because of that, tools for automatic
2 solution. Furthermore, it features 4 K ISP, native 16-bit model quantization and high levels synthesis were

DECEMBER 2020 IEEE A&E SYSTEMS MAGAZINE 47

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Towards the Use of Artificial Intelligence on the Edge in Space Systems: Challenges and Opportunities

developed [35]. A benchmark between the different COTS Since safety is critical for space applications, owing to the
for DL is shown in[28], [29], and [30]. high cost of failure, more predictable approaches to AI are
generally preferred [24]. To minimize risks in EO missions,
eventual applications of AI could be limited to the payload
AN AI-FIRST APPROACH TO SPACE APPLICATIONS level to perform object detection/classification locally on sen-
sor data [38]. In this case, eventual failures of AI would only
affect the quality of data for the single payload, without being
CHALLENGES WHEN USING AI a risk to the entire satellite. For lower dependability applica-
While AI is being already successfully applied in space, it tions, the use of COTS processors such as Myriad 2 or such
is still confined to offline data processing and not adopted as COTS FPGAs (mostly high-performance Ultrascale+
“on the edge” inside the spacecraft themselves. The pri- series FPGA) can be envisaged in all those applications with
mary reason is the difficulty of porting DL networks to low-to-medium dependability requirements, provided that
hardware that predates the algorithms themselves and has the AI inference accelerator is supervised by a fault-tolerant
insufficient performance to do even basic inferencing. For engine. The last issue involves the training of deep networks.
example, the weights and topology of models necessary to Indeed, a primary problem concerns the availability of data-
provide sufficient accuracy are often too large for the sets for training and model evaluation, especially for missions
memory budget of satellites. In addition, the inference of featuring new equipment, including novel sensors, for which
many AI models is computationally intensive [36], since a dataset for DNN training does not exist. Furthermore, in
they require a high number of operations per second to view of its complexity, DNN training shall be performed on
respect the latency requirements typical of many onboard ground by leveraging cloud-based GPUs or more specialized
applications. This is generally not acceptable for many training hardware such as TPUs. These aspects pose a real
applications owing to the power consumption constraints concern on the usability of models trained before the launch
due to the difficulty of heat dissipation and the low power of satellites, whose training is not performed through the orig-
budget. Such problems could be mitigated by adopting inal satellite data. However, this problem is mitigated by the
dedicated AI platforms, such as the Myriad 2, described in possibility of reconfiguring models during the life of mis-
section “State of the Art of COTS SoCs for DNN sions, enabled by the use of modern COTS ASICs and by the
Inference,” whose architecture and technology process reduced dimensions of files necessary for their programming,
permits the implementation of complex CNN models with which is becoming compatible with the uplink bandwidth of
excellent tradeoffs between model complexity, processing small satellites, as described in details in the section
speed, and power consumption. Furthermore, memory bud- “Effecting Change Through the Application of AI.”
get problems can be mitigated through an effective network
selection and design strategies by performing model com-
pression. In that respect, research indicates that certain neu-
ral network models can be compacted without significant
EFFECTING CHANGE THROUGH THE APPLICATION OF AI
loss of accuracy. For instance, knowledge distillation can The use of AI, in particular deep learning techniques, gen-
be applied to a model with a smaller size starting from a erally leads to better results in remote sensing than previ-
pretrained model [37]. Furthermore, quantization and prun- ous approaches [38]. Moreover, the benefits due to the
ing techniques can be efficiently applied to compress the introduction of deep learning would not be linked only to
model [35], even improving its accuracy compared to the the results of the single payload, but improvements in flex-
original one [32]. Depending on the arithmetic representa- ibility would be possible for the entire satellite.
tion used, different hardware can be exploited. Indeed, To better explain such a concept, consider the classical
Jetson Nano and Myriad 2 natively support only 16-bit architecture of an EO satellite, shown in Figure 1. Data are
floating point models. Myriad X also supports 8-bit fixed generally accumulated in a mass-memory and transmitted
point arithmetic. Google Coral uses an 8-bit fixed point sys- to ground when the satellite enters the coverage-area of a
tolic array. FPGAs can be exploited to implement both dedicated ground station. Such an approach is called
floating point and fixed point architectures. However, bet- “bent-pipe” communication paradigm, which involves
ter performances and a lower power per inference are sending the data to the ground following a command
obtained using fixed point [32], [35], reaching their peak of transmitted by the specific ground station [39]. In the clas-
performances exploiting binarized convolutional neural sical approach, the raw image produced by the imager is
networks [30], [35]. A second reason that may have slowed processed by an FPGA interface. The latter is used to con-
the adoption of AI for onboard applications is the lack of vert the images into a proprietary format, which might be
confidence in the unpredictability of the approach. This exploited by the image compression system, generally
nondeterminism derives from the impossibility of de facto realized through an ASIC solution, owing to the required
testing the weights set resulting from training, performed performances. Compressed images are finally stored in
through a finite number of data, for all possible inputs. mass memory before being transmitted to a ground station.

48 IEEE A&E SYSTEMS MAGAZINE DECEMBER 2020

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Furano et al.

Figure 1.
Classical EO satellite architecture. According to the classical approach, data are compressed, and stored into the mass-memory, according to
a first-in–first-out approach, before being transmitted to ground.

Figure 2.
AI-based EO satellite architecture. The proposed approach moves AI on board spacecraft to prefilter data according to the edge computing
paradigm, permitting to select useful data.

Performing image compression on raw sensor data by the AI platform for classification, image compression
reduces the downlink bandwidth and data storage require- is removed from the pipeline and all the images are stored
ments but restricts flexibility as images can be manipulated in raw format in memory. Consequently, the FPGA is
only after being downloaded and may not be desirable in only exploited to interface the imager and to store the raw
the future given the resolution trends for modern imagers, image data in the memory. The main disadvantage of this
which are driving downlink bandwidth [3]. This growth in approach is the increased onboard storage requirement to
sensor data bandwidth is not matched by a corresponding support the inference. However, current missions already
increase in data download speeds. Even switching from fly solid-state mass memories in the range of
1 Tb [41]
X-Band to Ka-Band will only increase downlink band- with projected capacities in the 12/20 Tb range for future
width by 10 while hyperspectral imagers increase raw missions. Furthermore, the proposed approach requires a
data generation by 200 with respect to current platforms. file system to distinguish data in different files, enabling
Preprocessing imagery on-the-edge before downlinking to demand-driven download through dedicated file delivery
generate actionable metadata onboard has huge advantages protocols, such as CCSDS File Delivery Protocol (CFDP),
in many cases, for instance, 50% or more of EO imagery whose implementations are under evaluation to increase
consists of cloud, which can safely be discarded or selec- its performances [19]. This protocol allows a CFDP user
tively processed before downlink [24], [40]. Moreover, an to exchange files and messages with other users and to
improvement of the sensor might require a redesign of the perform filestore operations and enable the transmission
hardware, including the image compression ASIC. of specific files from the satellite to ground. Furthermore,
The introduction of AI onboard EO satellites will since CFDP permits the implementation of filestore opera-
usher in a new era of on-demand metadata generation and tions, it might be exploited to manage the storage on the
content adaptation enabling high-value applications satellite, e.g., eliminate unused files. The advance of AI
unforeseen before satellite launch. Figure 2 shows our pro- research for edge inference is driving the development of
posed model, which represents a possible AI-based EO compact deep learning models with minimal accuracy
satellite architecture. The main idea is to exploit the AI loss [37]. In addition, the reduced size of these models
platform to classify the images depending on their content. means it is feasible to train machine learning models on
To this end, each new image is stored in memory before ground and update the onboard AI models via the band-
being processed by the AI engine. The classification pro- width-constrained satellite uplink.
cess can then be exploited to produce metadata derived Additional advantages of this approach are as follows:
from the image content thus enabling new applications.
Such metadata can provide concise information over files 1) Denby and Lucia in [39] demonstrated that prepro-
content, permitting to download only meaningful files, cessing on-the-edge of data and distribution of the
with advantages in terms of downlink bandwidth. To calculation over satellite constellations of satellites
avoid the removal of features, which might be exploited (orbital edge computing) leads to advantages in

DECEMBER 2020 IEEE A&E SYSTEMS MAGAZINE 49

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Towards the Use of Artificial Intelligence on the Edge in Space Systems: Challenges and Opportunities

terms of latency, reliability, and scalability. In addi- the use of DNNs for the processing of hyperspectral images.
tion, it reduces the number of ground stations that In particular, HyperScout 2 will host CloudScout, a CNN-
would be needed in the future to provide downlink based cloud detection algorithm. For such an application,
data requirements compared to the classic “bent to metadata are represented by the decision cloudy/not-cloudy
pipe” communication paradigm. image performed by the CloudScout model, which permit to
discard cloud-covered images [24], [39], [40]. The inference
2) The information provided by file metadata can be
of the CloudScout DNN will be processed through the Myr-
exploited to improve the execution of tasks at the
iad 2 accelerator, whose usability for this mission is con-
platform level. An example is represented by Cloud-
firmed by its resistance to radiation demonstrated through a
detection [24], [40] algorithms, which permit the
specific derisk test campaign, whose results are detailed in
preventive detection of clouds on satellite images.
Section IV. Other applications whose execution on board
Cloud covered images can, therefore, be discarded
might lead to benefits are those concerning fault-detection
with advantages in terms of bandwidth for the entire
and management. The work described in [45] presents a
satellite platforms [39].
methodology based on CNNs that permits to detect faults on
3) Increased system flexibility of COTS AI platforms satellite images and if the entity of corruption is such to com-
and modularity of DNNs means satellite hardware promise the use for the end-users. As for cloud detection,
designers can decouple the AI applications software when the final application is not critical for the reliability,
from the hardware due to the increased computa- the metadata produced might lead to discarding the final
tional headroom, thus reducing the satellite images with advantages in terms of data savings. On the con-
design and deployment time and cost. Moreover, trary, for those applications requiring reprocessing the
this fact might permit to reuse the same satellite image, the detection of “slightly corrupted” images allows
for different tasks. saving time due to image processing pipeline [45]. In addi-
tion to those applications producing benefits at the platform
4) As stated in the section “Challenges When Using
level, there are cases for which the execution on board
AI”, a primary challenge for the usability of DNNs in
might take advantage of better responsiveness due to the
space is the potential lack of suitable datasets for
edge computing approach [39]. Indeed, the preventive
training. This is the case for a satellite mounting a
knowledge of the content of the images permits the prioriti-
new imager or new sensors. Despite it represents a
zation of the download depending on the content itself.
major problem, a possible mitigation strategy is rep-
This might be essential for applications such as fire detec-
resented by the combined use of DNNs and COTS.
tion [46], which permits to catch the presence of fire in an
Indeed, DNNs can be pretrained through synthetic
area by exploiting the satellite images, or oil-spill detec-
data that emulate satellite datasets. Afterward, thanks
tion [47]. For both these applications, download prioritiza-
to reconfigurability and flexibility of COTS AI and
tion would allow reducing the latency in the detection of a
DNNs, pretrained models can be replaced during the
potential disaster and, therefore, to decrease the entity of
life of missions by more performant networks, trained
consequent damages. Finally, the AI-based architecture
using the same satellite data, reducing the probability
might be potentially exploited for many different EO appli-
of eventual failures over time. Such an approach was
cations, such as surface terrain classification, anomaly
exploited for the training of CloudScout [42], which
detection, change detection, object detection, and
represents the first European in-orbit demonstrator of
others [38]. For these applications, preprocessing the image
DNNs. Such an algorithm will be launched on-board
on board does not produce an immediate advantage. How-
a 6U-CubeSat exploiting a novel imager, HyperScout
ever, it can be used as a first preprocessing to extract the
2 [40],[43]. To train the model, an emulated dataset
metadata and identify the image of interest, performing
exploiting Sentinel-2 data was used [44], appropri-
content-based download, mitigating bandwidth require-
ately preprocessed to emulate HyperScout 2 images.
ments [39]. In particular, in the case of classification
algorithms, metadata are simply the result of classifica-
POSSIBLE APPLICATIONS OF AI-BASED PARADIGM tion. On the contrary, for tasks which do not require to
perform classification, metadata contain additional basic
FOR EO SATELLITES information which summarizes the content of the image.
As described in the section “Effecting Change Through the For instance, for applications which require to perform
Application of AI,” the use of the proposed AI-based archi- image segmentation, such as scene classification, the AI
tecture for some applications might benefit for the entire sat- platform should provide the pixel-level classification
ellite platform, such as cloud-detection. These concepts will results and metadata, which might contain a summary of
be applied on board the HyperScout 2 satellite, which will the different land partitions, for instance, by giving their
be launched in the frame of the PhiSat initiative [40], [42], percentage of occupations or by indicating the land
[43]. CloudScout represents the first in-orbit demonstrator of which has the highest percentage of occupation.

50 IEEE A&E SYSTEMS MAGAZINE DECEMBER 2020

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Furano et al.

Table 1.

Myriad 2 Network Execution FPSy

Network Myriad 2*** FPS Network Myriad 2*** FPS

alexnet***(227 x 277) 12.08 googlenet-v1 9.34*** (224 x 224)

googlenet-v2*** (224 x 224) 7.24 googlenet-v4 1.46*** (224 x 224)
inception-resent-v2*** (224 x 224) 1.42 mobilenet-ssd 10.62***(300 x 300)
resnet-101*** (224 x 224) 2.69 resnet-152 1.79***(224 x 224)
resnet-50*** (224 x 224) 5.13 squeezenet1.0 14.18***(224 x 224)
squeezenet1.1*** (224 x 224) 28.29 tiny-yolo 9.58***(416 x 416)
vgg16*** (224 x 224) 1.4 vgg17 1.15*** (224 x 224)
y
For each network, the frame size used during the test is reported.

down to at least 28 nm [48]. Nevertheless, testing of

GIANT LEAP FOR SPACE ELECTRONICS
COTS devices in relevant radiation environments is typi-
cally the only method whereby the expected perfor-
MYRIAD 2 AI ENGINE mance of the device can be characterized prior to flight.
The Myriad 2 platform [31] can efficiently perform AI SoCs in general present specific challenges for radiation
inference using pretrained networks as well as many characterization campaigns compared to other silicon
computer vision tasks such as visual odometry, optical devices such as memories (see [6], [10], and [20] for
flow, mesh-based image warping, stereo pair matching, background). Their typical combination of processor,
and keypoint tracking. An indicative list of the infer- memory (RAM and cache), peripherals, and mixed-sig-
ence speeds for popular networks is given in Table 1. nal elements, and in the case of FPGAs look up tables
The speeds were computed using the Myriad 2 (NCS1) (LUTs) and reconfigurable blocks, mean that a range of
over USB3 and with networks compiled using the test probing techniques are required. Moreover, multi-
OpenVINO (R5) software stack and computed with the processor SoCs are inherently more difficult and time-
sample benchmark code, which is provided with the consuming to characterize due to their duplication of
publicly available OpenVINO software. Testing was certain elements (e.g., clock managers, caches), their
carried out by the authors. Importantly, these inference sharing of larger memories, and their more complicated
performance speeds are all achieved within a 1.2-W datapaths.
power envelope. Significant work on radiation effects on FPGA-
based systems has been conducted under the auspices of
the ALICE project within the umbrella of CERN
research [49], where parts of the collider sensing equip-
RADIATION TESTING OF COTS DEVICES ment are managed by Xilinx Virtex-II COTS FPGA-
Like all candidate hardware to be flown in space, Myriad 2 controlled subsystems that are exposed to high energy
first needs to be tested for radiation tolerance as space is particles. Assessment of SEEs in this FPGA helps us to
full of charged particles from the Sun and further out in predict the downtime of the sensing equipment. At
the cosmos. These particles can cause SEEs within the sil- 28 nm, several radiation campaigns have been con-
icon structure of the device, results in bit flips (SEUs), ducted to assess the performance of the Xilinx Zynq
latch-ups (SELs), and functional interrupts (SEFIs) in the 7000 COTS SoC wherein testing was conducted with
device. Prolonged exposure to such particles can also heavy ions [50],[51]. Recently, guidelines specifically
cause failure due to TID. addressing the radiation testing of SoC devices have
The process technology node on which the device highlighted the relative lack of experience in the space
is manufactured can indicate likely error rates, inde- community in testing multicore SoCs [52].
pendent of testing. For example, SEU rates of embed- At a practical level, a complete SEL test setup for
ded memories have been seen to decrease from 180 to COTS devices is typically made more difficult due to
60 nm, but SRAM has been shown to display rising the often large number of power rails to be monitored
SEU rates below 40 nm [48]. On the other hand, flop and the variation (sometimes by order of magnitude)
error rates decrease as technology nodes decrease of the expected/maximum current on each rail.

DECEMBER 2020 IEEE A&E SYSTEMS MAGAZINE 51

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Towards the Use of Artificial Intelligence on the Edge in Space Systems: Challenges and Opportunities

Figure 3.
ESA/Ubotica teams performing radiation testing of Myriad 2 at CERN.

SEUs are measured at bit level for memories via a

INITIAL STEP FOR MYRIAD 2 - DERISKING RADIATION
series of static analysis tests, one for each memory (includ-
CHARACTERIZATION ing L1 and L2 caches for each processor) in the device.
Myriad 2 enables the caches to be disabled and then
The Myriad 2 SoC has unique characteristics that make its directly written—test patterns equal to the word address
radiation testing challenging, such as its 14 processor cores, are written to the two RAM memories in the device [DDR
its multiported shared memory, and its stacked processor and Connected MatriX memory (CMX)]. Cache tests are
and memory dies. Indeed, to the authors’ knowledge, the conducted in isolation so as to decorrelate effects. SEFIs
Myriad 2 is one of the most complex SoCs ever to undergo are exposed by analyzing failure rates in representative
radiation characterization for space applications. The fact functional tests for data movement from/to CMX memory
that Myriad 2 is a SiP rather than a monolithic chip means via the SHAVE processing cores, for a bilinear image
that highly penetrating ion beams have to be used to pierce resize task implemented as vector operations on the
the 1.5-mm-thick package and the stacked double data rate SHAVEs and using the DMA engine, and for cache read/
(DDR) die. Such beams were recently utilized during Myr- write cycle tests. Individual SIPP filters are functionally
iad 2 radiation test campaigns at CERN (France) and GSI exercised via image processing unit tests, enabling input
(Germany). In the case of CERN, a special agreement gave and output images to be directly compared. Over 3000
ESA access to the most intense beam of ultrahigh energy Myriad 2 registers are automatically downloaded from the
heavy ions available—short of traveling into orbit. device at the end of every test.
In preparation for the SEE test campaigns at these two A dedicated jig, seen in Figure 3, was designed to
facilities, a set of unique software tests was developed to expose two test boards for the CERN campaign, allowing
exercise the gamut of functionalities of the SoC and to ren- both SEL and SEU testing to be conducted simulta-
der all errors and erroneous behavior visible. Dynamic neously. The SEL board was biased and executed a basic
memories, all six caches, a selection of six of the SIPP image heartbeat program to enable monitoring. Myriad 2 sup-
processing hardware filters, the direct memory access ply rails were grouped to common nominal voltages
(DMA) engine, and the 12 SHAVE vector processors are for the test, with each voltage supplied and monitored
exercised in the test suite. Memory and cache patterns are for latch-up events by a benchtop power supply with
captured within static tests for posttest analysis, enabling integrated current monitoring and overcurrent shut-
multibit upsets, and nonstochastic errors to be identified. down. The current limit for each group of rails was set
The software test set focused on assessing the internal archi- according to the maximum current measured for that
tecture of the device rather than activating the device periph- group prior to irradiation, across the entire set of tests.
erals, with JTAG the only interface utilized (for initiating The Myriad 2 case was conductively heated to a stable
and monitoring tests and downloading results). 68 C case temperature.

52 IEEE A&E SYSTEMS MAGAZINE DECEMBER 2020

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Furano et al.

Figure 4.
Myriad 2 functional cross-sections across all functional tests, at LET=2.2 MeVcm2/mg.

Initial testing was conducted with lead ions at the H8 SEE mitigation strategy. Apart from the SHAVE L1 cache,
beamline of CERN’s Super Proton Synchrotron (SPS) at a all caches have bit cross-sections that are within approxi-
Linear Energy Transfer (LET) of 8.8 MeVcm2/mg (Si). A mately an order of magnitude of each other, indicating that
further test campaign at the SIS high-energy beamline at the decision on whether or not to enable caches can be
GSI provided a second round of testing in which a single made at the device level. The CMX bit cross-section sug-
Myriad 2 was bombarded with Fe ions at a LET of gests that it can be viewed as a “super-cache” in terms of its
2.2 MeVcm2/mg (Si). SEFIs and SEUs were captured dur- susceptibility to radiation and in terms of its usage (i.e., it
ing this campaign. should mainly be used for transitory storage). The bit upset
For both tests, extensive posttest analysis was necessary patterns within the LEON caches appeared random, and
to understand the behavior of the SiP under the intense radi- the ratio of 1!0 and 0!1 bit flips are in statistical agree-
ation beams. Figure 4 presents the SEFI results from the ment with the distributions of 0 and 1 s within the test pat-
GSI campaign. The functional cross-section shown in the terns. The SHAVE L1 cache was susceptible to nonrandom
plot indicates the rate at which SEFIs occurred as a function bit upsets, which greatly increased its bit cross-section.
of the ions per device (normalized by the die area). Results While the cause of these errors is under investigation, it
are averaged across all tests within the same functional may relate to SETs on the cache address lines.
type. The plot illustrates that the functional cross-sections A 60 Co TID irradiation test was conducted at ESA‘s
60
across all tests are within an order of magnitude of each Co test facility in ESTEC, where two Myriad 2 devices
other, providing overall device functional cross-section at a on identical test boards were tested in parallel while
2.2 MeVcm2/mg LET. The SIPP tests in general have larger exposed to ionizing radiation at a rate of 372 rad/h. After
functional cross-sections than the remaining tests, but the a total dose of 49 krad (Si) both devices were still fully
size of these tests’ code sections may be a factor in their functional and the test results showed no errors on either
apparent higher cross-sections (greater probability of code device during the course of the campaign. Current meas-
corruption). The SIPP test results indicate that the SIPP fil- urements gathered during the tests did not show any
ters are susceptible to silent data corruptions and that such upward trend in current draw, indicating no abnormal
occurrences can induce “block” errors in their computa- operation of the devices during or after irradiation. The
tional output. mean device internal junction temperatures increased by
The bit cross-sections across all memories and caches less than 0.6 C over the course of the 133-h Co-60 test.
on the Myriad 2, and at both tested LET, are shown in Further testing of the same Myriad 2 s was conducted sub-
Figure 5. Each data point is averaged across all tests within sequent to the Co-60 test in order to investigate the effects
the same type and normalized by the die area. It can be of autoannealing on the Myriad 2. Both devices were fully
seen that the DDR bit cross-section is several orders of functional after continuous functional testing for seven
magnitude lower than that of CMX, suggesting storing and days at room temperature. All intermediate and final tests
running code from DDR rather than CMX as a possible were performed at maximum frequency, and no residual

DECEMBER 2020 IEEE A&E SYSTEMS MAGAZINE 53

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Towards the Use of Artificial Intelligence on the Edge in Space Systems: Challenges and Opportunities

Figure 5.
Myriad 2 normalized bit cross-sections across all memories.

effects on Myriad 2 functionality were observed during or CONCLUSION

after the TID campaign.
The primary results from the derisking radiation tests Space embedded systems are significantly behind commer-
are that Myriad 2: cial ones due to the high cost and long design times of
space-qualified components. Earth Observation applica-
 is functionally unaffected by a 49 krad TID;
tions, especially in the emerging low-cost constellation
 had no confirmed latch-up events at a fluence of market, are ideally positioned to leverage state-of-the-art
1.045106 ions/device and a LET of 8.8 MeVcm2/mg; power-efficient onboard data processing capabilities that
 has an in-package DDR that is significantly less sen- enable AI inference on-board, to increase the imaging
sitive to ionizing bit upset effects than its other product value and to make optimum use of always limited
memories, while its CMX memory has cross-section downlink bandwidth. Recent advances in space avionics to
approximately equivalent to that of its caches; decentralized on-board compute mean that COTS edge
processors are also ideally positioned to deliver low-
 has a measured functional cross-section that is latency distributed edge compute at source enabling new
within an order of magnitude across all functional low-latency value-added services from orbit. Furthermore,
tests, as measured at a LET of 2.2 MeVcm2/mg. rapid mission design cycles are possible using COTS devi-
The results indicate that Myriad 2 does not have critical ces that incorporate suitable mitigation strategies, and mis-
sensitivity to radiation at low LETs, further indicating its sion lifetime extensions are immediately feasible for AI
potential for in-orbit application and progressing its path to solutions by dynamic reconfigurability of the neural net-
flight. These results are being used to inform software miti- works. The Intel Myriad 2 VPU is a best-in-class COTS
gation strategies for a number of planned in-orbit demon- computer vision and AI engine for edge applications with
strators that integrate Myriad 2 as an AI inference engine. extremely high processing efficiency, high streaming
A future Myriad 2 radiation characterization campaign, throughput, small size, and low-cost making it ideally suited
conducted across a broader range of LETs, is planned for for the ultimate edge application: space! A radiation charac-
H1 2020, enabling Weibull SEU curve fitting and LET terization programme for Myriad 2 has recently been con-
threshold determination. Functional AI inference tests ducted to assess the device‘s radiation tolerance in the
across a range of popular networks, and including the mon- context of space deployments and to facilitate the design of
itoring of neural patterns at internal network layers, will be software mitigation strategies. Results from the derisking
conducted during this future test campaign, providing fur- test campaigns indicate that Myriad 2 is inherently robust to
ther insight to the community on the radiation effects on AI latch-up effects at the LET tested, and is immune to TID
inference silicon devices. effects up to 49 krad. Its selection of memories and caches

54 IEEE A&E SYSTEMS MAGAZINE DECEMBER 2020

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Furano et al.

have cross-sections that, with careful software architecture [13] L. Summerer, “Evaluating research for disruptive innova-
and memory mapping, can facilitate robust solutions in the tion in the space sector,” Acta Astronautica, vol. 81, no. 2,
context of a COTS device. Further radiation and environ- pp. 484–498, 2012.
mental characterization tests are in the pipeline on the path [14] L. Sterpone, S. Azimi, and B. Du, “A selective mapper for
to Myriad 2’s qualification for deployment in planned the mitigation of SETs on rad-hard RTG4 flash-based
orbital missions as a computer vision and AI edge processor FPGAs,” in Proc. IEEE 16th Eur. Conf. Radiat. Effects
for EO and avionics applications. Compon. Syst., 2016, pp. 1–4.
[15] R. Hillman, G. Swift, P. Layton, M. Conrad, C. Thibodeau,
and F. Irom, “Space processor radiation mitigation and
validation techniques for an 1,800 mips processor board,”
REFERENCES
in Proc. 7th Eur. Conf. Radiat. Effects Compon. Syst.,
[1] “The future of the European space sector,” Jun. 18, 2020. 2003, pp. 347–352.
Accessed: Jul. 24, 2020. [Online]. Available: https://www. [16] “GR740: The ESA Next Generation Microprocessor
eib.org/attachments/thematic/future_of_european_space_ (NGMP),” Jun. 18, 2020. Accesed: Jul. 24, 2020. [Online].
sector_en.pdf Available: http://microelectronics.esa.int/gr740/index.html
[2] M. N. Sweeting, “Modern small satellites-changing the [17] “ArianeGroup and PTScientists to study lunar lander mis-
economics of space,” Proc. IEEE, vol. 106, no. 3, sion for ESA,” Jan. 22, 2019. Accessed: Jul. 24, 2020.
pp. 343–361, Mar. 2018. [Online]. Available: https://spacenews.com/arianegroup-
[3] P. Wertz, B. Hespeler, M. Kiessling, and F.-J. Hagmanns, and-ptscientists-to-study-lunar-lander-mission-for-esa/
“Next generation high data rate downlink subsystems [18] “ESA/Roskosmos LUNA mission,” Accessed: Jul. 24, 2020.
based on a flexible APSK modulator applying SCCC [Online]. Available: http://www.esa.int/Science_Exploration/
encoding,” in Proc. IEEE Int. Workshop Tracking, Teleme- Human_and_Robotic_Exploration/Exploration/Luna
try Command Syst. Space Appl., 2016, pp. 1–7. [19] G. Meoni, A. Valverde, G. Magistrati, and L. Fanucci,
[4] D. Selva and D. Krejci, “A survey and assessment of the “Estimating the downlink data-rate of a CCSDS file deliv-
capabilities of cubesats for earth observation,” Acta Astro- ery protocol IP core,” in Proc. Int. Conf. Appl. Electron.
nautica, vol. 74, pp. 50–68, 2012. Pervading Ind., Environ. Soc., 2019, pp 513–519.
[5] W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, “Edge com- [20] A. Menicucci et al., “Simplified procedures for COTS TID
puting: Vision and challenges,” IEEE Internet Things J., testing: A comparison between 90Sr and 60Co,” in Proc.
vol. 3, no. 5, pp. 637–646, Oct. 2016. IEEE Radiat. Effects Data Workshop, 2018, pp. 1–6.
[6] G. Lentaris et al., “High-performance embedded comput- [21] N. P. J., et al., “In-datacenter performance analysis of a
ing in space: Evaluation of platforms for vision-based nav- tensor processing unit,” in Proc. ACM/IEEE 44th Annu.
igation,” J. Aerosp. Inf. Syst., vol. 15, no. 4, pp. 178–192, Int. Symp. Comput. Archit., Jun. 2017, pp. 1–12.
2018. [22] R. Ginosar, “Survey of processors for space,” Eur. Space
[7] P. Fortescue, G. Swinerd, and J. Stark, Spacecraft Systems Agency (Special Publication) ESA SP, vol. 701, 2012.
Engineering. Hoboken, NJ, USA: Wiley, 2011. [23] “Next-generation landing technology,” Accessed: Jul. 24,
[8] P. Manzano et al., “Heavy ion latch-up test on dsPIC micro- 2020. [Online]. Available: https://www.esa.int/Science_
controller to be used in ExoMars 2020 mission,” in Proc. Exploration/Human_and_Robotic_Exploration/Lunar_
IEEE Radiation Effects Data Workshop, 2017, pp. 1–4. Lander/Next-generation_landing_technology
[9] L. Ding, S. Gerardin, M. Bagatin, D. Bisello, S. Mattiazzo, [24] V. Kothari, E. Liberis, and N. D. Lane, “The final frontier:
and A. Paccagnella, “Investigation of total ionizing dose Deep learning in space,” in Proc. 21st Int. Workshop
effect and displacement damage in 65 nm CMOS transistors Mobile Comput. Syst. Appl., 2020, pp. 45–49.
exposed to 3 MeV protons,” Nuclear Instrum. Methods [25] P. Blacker, C. Bridges, and S. Hadfield, “Rapid prototyp-
Phys. Res. Sec. A, Accelerators, Spectrometers, Detectors ing of deep learning models on radiation hardened CPUs,”
Associated Equip., vol. 796, pp. 104–107, 2015. in Proc. IEEE NASA/ESA Conf. Adaptive Hardware Syst.,
[10] G. Furano and A. Menicucci, “Roadmap for on-board proc- 2019, pp. 25–32.
essing and data handling systems in space,” in Dependable [26] X. Wang, Y. Han, V. C. Leung, D. Niyato, X. Yan, and
Multicore Architectures at Nanoscale. New York, NY, X. Chen, “Convergence of edge computing and deep
USA: Springer, 2018, pp. 253–281. learning: A comprehensive survey,” IEEE Commun. Surv.
[11] M. P. Baze, S. P. Buchner, and D. McMorrow, “A digital Tuts., vol. 22, no. 2, pp. 869–904, Apr.–Jun. 2020.
CMOS design technique for SEU hardening,” IEEE Trans. [27] “AI flood drives chips to the edge,” Nov. 7, 2018. Accessed:
Nuclear Sci., vol. 47, no. 6, pp. 2603–2608, Dec. 2000. Jul. 24, 2020. [Online]. Available: https://www.eetimes.
[12] N. Rezzak, E. X. Zhang, M. L. Alles, R. D. Schrimpf, and com/ai-flood-drives-chips-to-the-edge/
H. Hughes, “Total-ionizing-dose radiation response of [28] W. Li and M. Liewig, “A survey of AI accelerators for
partially-depleted SOI devices,” in Proc. IEEE Int. SOI edge environment,” in Proc. World Conf. Inf. Syst. Tech-
Conf., 2010, pp. 1–2. nol., 2020, pp. 35–44.

DECEMBER 2020 IEEE A&E SYSTEMS MAGAZINE 55

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.
 Towards the Use of Artificial Intelligence on the Edge in Space Systems: Challenges and Opportunities

[29] M. Antonini, T. H. Vu, C. Min, A. Montanari, A. Mathur, [41] M. Fabiano and G. Furano, “NAND flash storage technol-
and F. Kawsar, “Resource characterisation of personal- ogy for mission-critical space applications,” IEEE Aerosp.
scale sensing models on edge accelerators,” in Proc. 1st Electron. Syst. Mag., vol. 28, no. 9, pp. 30–36, Sep. 2013.
Int. Workshop Challenges Artif. Intell. Mach. Learn. Inter- [42] M. Esposito, “CloudScout: In orbit demonstrationof
net Things, 2019, pp. 49–55. machine learning applied on hyperspectral and multispec-
[30] E. Wang et al., “Deep neural network approximation tral thermal imaging,” European Space Agency, 2019.
for custom hardware: Where we’ve been, where we’re [43] M. Esposito, S. Conticello, M. Pastena, and
going,” ACM Comput. Surv., vol. 52, no. 2, pp. 1–39, B. C. Dom
ınguez, “In-orbit demonstration of artificial intel-
2019. ligence applied to hyperspectral and thermal sensing from
[31] B. Barry et al., “Always-on vision processing unit for space,” Proc. SPIE, vol. 11131, 2019, Art. no. 111310C.
mobile applications,” IEEE Micro, vol. 35, no. 2, pp. 56–66, [44] M. Drusch et al., “Sentinel-2: ESA’s optical high-resolution
Mar./Apr. 2015. mission for GMES operational services,” Remote Sens.
[32] G. Dinelli, G. Meoni, E. Rapuano, G. Benelli, and Environ., vol. 120, pp. 25–36, 2012.
L. Fanucci, “An FPGA-based hardware accelerator for [45] M. Biasielli, C. Bolchini, L. Cassano, E. Koyuncu, and
CNNs using on-chip memories only: Design and benchmark- A. Miele, “A neural network based fault management
ing with intel movidius neural compute stick,” Int. J. Recon- scheme for reliable image processing,” IEEE Trans.
figurable Comput., vol. 2019, 2019, Art. no. 7218758. Comput., vol. 69, no. 5, pp. 764–776, May 2020.
[33] S. Mittal, “A survey of FPGA-based accelerators for convo- [46] B. Hally, L. Wallace, K. Reinke, S. Jones, and A. Skidmore,
lutional neural networks,” Neural Comput. Appl., vol. 32, “Advances in active fire detection using a multi-temporal
pp. 1109–1139, 2020. method for next-generation geostationary satellite data,”
[34] G. Dinelli, G. Meoni, E. Rapuano, and L. Fanucci, Int. J. Digit. Earth, vol. 12, no. 9, pp. 1030–1045, 2019.
“Advantages and limitations of fully on-chip CNN FPGA- [47] M. Krestenitis, G. Orfanidis, K. Ioannidis, K. Avgerinakis,
based hardware accelerator,” in Proc. IEEE Int. Symp. Cir- S. Vrochidis, and I. Kompatsiaris, “Early identification of
cuits Syst., to be published. oil spills in satellite images using deep CNNs,” in Proc.
[35] M. Blott et al., “FINN-R: An end-to-end deep-learning Int. Conf. Multimedia Modeling, 2019, pp. 424–435.
framework for fast exploration of quantized neural [48] A. Dixit and A. Wood, “The impact of new technology on
networks,” ACM Trans. Reconfigurable Technol. Syst., soft error rates,” in Proc. IEEE Int. Rel. Phys. Symp.,
vol. 11, no. 3, pp. 1–23, 2018. 2011, pp. 5B.4.1–5B.4.7.
[36] G. Benelli, G. Meoni, and L. Fanucci, “A low power key- [49] J. Alme et al., “RCU2—The ALICE TPC readout elec-
word spotting algorithm for memory constrained embed- tronics consolidation for Run2,” J. Instrum., vol. 8, no. 12,
ded systems,” in Proc. IFIP/IEEE Int. Conf. Very Large pp. C12 032–C12 032, 2013.
Scale Integr., 2018, pp. 267–272. [50] L. Tambara, A. Akhmetov, D. Bobrovsky, and
[37] G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, F. Kastensmidt, “On the characterization of embedded mem-
“Learning efficient object detection models with knowl- ories of Zynq-7000 all programmable SoC under single event
edge distillation,” in Proc. Adv. Neural Inf. Process. Syst., upsets induced by heavy ions and protons,” in Proc. IEEE
2017, pp. 742–751. 15th Eur. Conf. Radiat. Effects Compon. Syst., 2015, pp. 1–4.
[38] X. X. Zhu et al., “Deep learning in remote sensing: A [51] L. Tambara et al., “Heavy ions induced single event upsets
comprehensive review and list of resources,” IEEE Geosci. testing of the 28 nm Xilinx Zynq-7000 all programmable
Remote Sens. Mag., vol. 5, no. 4, pp. 8–36, Dec. 2017. SoC,” in Proc. IEEE Radiat. Effects Data Workshop,
[39] B. Denby and B. Lucia, “Orbital edge computing: Nanosa- 2015, pp. 1–6.
tellite constellations as a new class of computer system,” [52] S. M. Guertin, “Guideline for single-event effect
in Proc. 25th Int. Conf. Archit. Support Program. Lang. (SEE) testing of system on a chip (SOC) devices,” Feb. 1,
Oper. Syst., 2020, pp. 939–954. 2018. Accessed: Jul. 24, 2020. [Online]. Available: https://
[40] M. Esposito et al., “Hyperscout-2: Highly integration of ntrs.nasa.gov/search.jsp?R=20190002148, 2018
hyperspectral and thermal sensing for breakthrough in-
space applications,” in ESA Earth Observation f-Week,
Sep. 9–13, 2019, ESRIN, Frascati (ROMA), Italy.

56 IEEE A&E SYSTEMS MAGAZINE DECEMBER 2020

Authorized licensed use limited to: Tsinghua University. Downloaded on December 19,2020 at 11:21:30 UTC from IEEE Xplore. Restrictions apply.