ECEE 5623 - Real-Time Embedded Systems, ESE Program

Class: ECEE 1B32, Wednesdays, Flipped Classroom (ECEE 1B32) using Zoom, May 29th until
August 16, 2019 - During Term-D (June 3 to Aug 9).
Final project demonstration day - August 15, Dr. Siewert available for demos all day on campus
(8/17 early morning for overflow).

Join @ 4:00PM Colorado Time Wednesdays with ZOOM (Flipped Class)

DEBUG or DEMO of Help Sessions with SA ZOOM

Please watch Video Lectures and consult Lecture Notes before each flipped class. Please use
one of the supported embedded platforms for the course along with supporting tools as
outlined here to complete Labs and turn them in on Canvas [Canvas].

Instructor: Sam Siewert, [email protected], [email protected]

Office Hours: Available by e-mail request.
SkypeID: sam.siewert
Office (voice): 928-777-6929
Cell (Text preferred): 303-641-3999

Graduate Student enrollment and advising - please e-mail Adam Sadoff

([email protected]) with questions.

SA, Grader: Mohit Rane, [email protected]

Office Hours: Mondays and Wednesdays 2-4 PM (on campus).
Skype: live:mrane2010
Phone (Text preferred): 720-789-1750

SA, Grader: Shreya Chakraborty, [email protected]

Office Hours: Tues & Thursday 1:30-3:30 PM.
Skype: live:shreya12ee1037
Phone (Text preferred): 720-436-8127

Course Prerequisites: Strong C programming skills (some C++ helpful), familiarity with

operating systems, computer architecture and hardware/software interface and debug skills
are required. Advanced skills and knowledge of embedded CPU multi-core, GP-GPU, and/or
FPGA are helpful, but not required. Before taking ECEE 5623, students should have a first
course in related subjects such as ECEN 2120/2350, ECEN 3100/3350, and ECEN
1030/1310/CSCI 1300 and/or have completed ECEN 5803 and ECEN 5813 prior to taking this
course. Real-time theory will be taught using embedded Linux or an equivalent RTOS or IoT
kernel and students will be expected to work with camera interfaces and real-time sensor
systems to build a verifiable predictable response application.
Course Description: In this course, students will design and build a microprocessor-based
embedded system application using a real-time operating system or RT POSIX extensions with
Embedded Linux. The course focus is on the process as well as fundamentals of integrating
microprocessor-based embedded system elements for digital command and control of typical
embedded hardware systems. Standard project options include: stereo vision, computer vision,
voice-over-IP, computer vision tracking systems, facial recognition and numerous related
projects. The student will be introduced to the full embedded system lifecycle process in this
course including: analysis, design (using extended finite state machine specification tools),
programming, hardware assembly, unit testing, integration and system testing.

Lab Description: The course requires the student to install embedded Linux or an RTOS
microkernel (FreeRTOS or Zephyr) on a System-on-Chip processor such as the Raspberry-Pi 3b+,
NVIDIA Tegra K1 or Nano, the Altera DE1-SoC, or the Intel Curie / Quark (Arduino 101). In
addition, students may want to set up an Oracle Virtual-Box Linux installation. For work with
live video please note recommended cameras and make sure you have access to native JetPack
Linux on a Jetson (TK1 or Nano). For other types of sensor interfaces and audio, the DE1-SoC or
Curie/Quark is fine. This course must be completed using an embedded system RTOS (FreeRTOS
or Zephyr) or embedded Linux only, not a PC running Linux. You will however find Linux as a
useful host development system (Windows is ok too).

1. REQUIRED TEXT: Real-Time Embedded Components and Systems with Linux and RTOS,
Sam Siewert and John Pratt, December 2015, 978-1942270041, Mercury
Learning, Amazon, E-book, RTECS 2nd Ed Figures and Examples, RTECS 1st Ed
CDROM, ERRATA
2. Linux Kernel Development (3rd Edition), Robert Love, Addison-Wesley Professional; 3
edition (July 2, 2010), ISBN-10: 0672329468, ISBN-13: 978-0672329463, Linux Kernel
Development (3rd Edition), by Robert Love

Recommended platforms for the class (in order of ease of use):

1. Raspberry Pi 3 model B from Amazon Starter Kit described on eLinux,

2. Buy Jetson Nano and Get Started with Nano, Jetson Nano eLinux, or Jetson
TK1 from NVIDIA embedded using L4T.
3. Altera DE1-SoC which can run Linux or FreeRTOS,
4. Intel Curie/Quark SoC running Zephyr microkernel Intel Curie/Quark (Arduino 101) is a
good option for Zephyr Development using the FlySwatter2 to flash the board
and Adafruit FTDI cable for a console.
5. Embedded Debian Linux on TI-OMAP (Being Phased Out, but OK) Beagle xM or Beagle
Bone Black (Debian likely pre-installed),
6. Host Development and Test on Native Linux Ubuntu LTS (12.04 or 14.04) download and
install on any x86 laptop/desktop,
 7. Host Development and Test with Oracle Virtual Box on Windows or Mac OS-X with
Ubuntu LTS installation on VM (note that you can only process pre-recorded MPEG), 5)
Use of MobaXterm or Xming/Putty and ECES server with ecee-gpu4 or ecee-gpu5. For
Linux, once you have an Ubuntu platform set up, (excluding ECES server), you'll want to
install OpenCV as described here and more generally here. If you have a Linux laptop,
you can most likely use your built-in webcam (test this with class example code
found here) or a simple UVC supported external webcam such as a Logitech C270. NOTE
THAT YOU MUST COMPLETE YOUR EXTENDED LAB on an EMBEDDED Jetson TK1 or
Nano, DE1-SoC or Beagle Board and that other Linux installations are intended to be
used only for practice and support - i.e. you must use an embedded Linux or RTOS
system to complete the course.

Remote Lab Note: Embedded Linux, Linux Foundation Zephyr or FreeRTOS are ALL acceptable

platforms for this course and there are numerous options for your lab setup. Please review
and purchase and configure the option that works best for you. The ESE Program office DOES
HAVE SOME LOANER SYSTEMS you can check out as well - see your TA for help if you prefer this
option. Note that the NVIDIA Jetson TK1 or Nano can also be used for ECEE57663 (EMVIA) and
note that the DE1-SoC has an FPGA which may be used in other embedded courses. For Zephyr,
the Intel Curie/Quark (Arduino 101) is a good option for Zephyr Development using
the FlySwatter2 to flash the board and Adafruit FTDI cable for a console.

Important Course Links (Linux, RTOS=VxWorks, FreeRTOS, Zephyr)

Lab Exercises
Video Lectures
Linux Example Code, Linux Example Code 2, RTOS Example Code, FreeRTOS Example Code
Linux Docs, RTOS Docs, FreeRTOS Docs, Linux Top-Errors, RTOS Top-Errors, 5623 Classic
Papers Read in Class
Course and Grading Policies

Important Course Tools

DE1-SoC Tools: DE1-SoC Board Terasic Page, ARM DS-5 Community Edition, Altera FTP
Site, Altera DE1-SoC Labs, DE1-SoC Resources, Altera SoC EDS Downloads, Quartus II FPGA
Design Tools,Class Documents for DE1-SoC.
NVIDIA Jetson Tools: NVIDIA Jetson Page, Jetson TK1 eLinux Page, Jetson Nano Page, Class
Documents for Jetson.
TI Beagle Board Tools: Beagle Board Page, Class Documents for Beagle.
Intel Curie/Quark Tools: Curie Quark How-To, Linux Foundation Zephyr for Quark.
TIVA FreeRTOS Tools: Class Documents for TIVA FreeRTOS.
OpenCV
Oracle Virtual-Box
Ubuntu Linux LTS, 14.04 LTS Download
Linux UVC Supported Cameras, e.g. Logitech C200, or for Analog Camera Frame
Grabber, Hauppauge USB-Live 2 used with any low-cost NTSC CCTV camera, e.g. Wired Color
CCTV Security Camera - cost ranging from $10 up to $50.

Syllabus (See Canvas for Definitive Lab Due Dates and Quizzes) Term-D - June to August

Week-1, Session-1 Class: Introduction - Lecture 1 Video , Lecture 1 Notes , Updated Notes
- Course goals [10 weeks, 5 labs, Mid-Term Exam, 2 Quizzes]
* Beagle xM and ARM DM architecture [ARM v7te ISA, Coretex A8]
* Ubuntu Linux
* Linux user space and kernel space programming
* Hard Real-Time Theory
* Drivers and Real-Time Video
* 1/3 Theory, 1/3 Practice, 1/3 Practitioner Work
- Course project [Final Extended Lab - USB Camera for Time Lapse Capture]
- Course grading (see policy below)
- Fall ECEE 5623 Video Files: Project from Previous Semesters (Robotics, Video, Audio)
- RT Embedded System TERMINOLOGY
INITIAL QUIZ REVIEW - Terminology Overview

Linux Host Development System setup (Tips on Beagle xM or NVIDIA Jetson)

- Booting Ubuntu and other Linux Distros on Beagle xM
- Native host Linux (Ubuntu) or Oracle Virtual-Box Linux on Windows
- See Nisheeth Bhat's IS on TimeSys for Hard Real-Time
- Kaushik's Instructions for loading Ubuntu on Beagle xM
- Beagle xM
- "cat /proc/devices"
- lsmod
- dmesg
- "tail -f /var/log/messages"
- ECEE 5623 Example Linux Code

Paper: Liu and Layland Rate Monotonic paper

Read: RTECS (Real-Time Embedded Components and Systems) - Chapter 1

Start Lab-1

Week-1, Session-2 Class: Fundamentals of Real-Time Theory - Lecture 2 Video , Lecture 2

Notes , Updated Notes
- What's makes distributed, real-time, embedded multitasking systems challenging?
- Best effort, Hard real-time, Soft real-time
- Scheduling Class Taxonomy
- Best Effort scheduling (Round-Robin Timeslice Scheme - Review)
- Intro to Fixed priority preemptive scheduling
- Intro to Dynamic priority scheduling
- Utility Curves

Read: RTECS - Chapter 2

Paper: Lehoczky Sha and Ding paper

Week-2, Session-1 Class: Fundamentals of RT Theory Cont'd - Services and High-Level Design -
QUIZ on Terminology Overview, Lecture 3 Video , Lecture 3 Notes , Updated Notes

INITIAL QUIZ - On Terminology - Complete on Canvas

Real-Time Services
- What a Service Release Timeline Really Looks Like
- The CPU, I/O, Memory Resource Space (Characterizing RT Applications)
- Intro to Timing diagrams (interference)
- Intro to Hard real-time safe resource utilization bounds

Case-Study #1: High-Level RT System Design Examples

EXAMPLE - Space Infrared Telescope Facility Instrument Control (launched on August 25,
2003)
- The hard real-time requirements and performance
- What worked, what didn't and why
- Overview Presentation
- The Test Image

EXAMPLE - DATA-CHASER Space Shuttle Hitch-hiker Class Payload Flown on STS-85,

Summer 1997

Paper: Audsley DM paper

INITIAL QUIZ - On Terminology - Complete on Canvas

Lab-1 DUE
Start Lab-2

Week-3, Session-1 Class: RM/DM Policy and Feasibility Test Derivation - Lecture 4 Video ,
Lecture 4 Notes , Updated Notes

Rate Monotonic Policy and Feasibility Overview

- Rate Monotonic Assumptions and Constraints
- More on Fixed priority preemptive scheduling
- Overview of Derivation of RM LUB - Hard real-time safe resource utilization bounds
- More on Dynamic priority scheduling (Earliest Deadline First and Least Laxity First)
- EDF and LLF Overview

Introduction to Feasibility Tests

- Sufficient Tests - RM LUB, DM
- N&S Tests
- Scheduling Point
- Completion Test

Deadline Monotonic Policy and Feasibility Overview

- DM Sufficient Feasibility Test
- DM Sufficient and More Necessary Test

Beyond RM and DM:

- Scheduling Point Feasibility Test
- Completion Test Feasibility Test
- Sufficient vs. Necessary and Sufficient Feasibility Tests
- Computational Complexity of Feasibility Tests (O(1), O(n^2), O(n^3))

PROJECT BACKGROUNDER: COMPUTER VISION, Video-Lecture-Machine-Vision, PDF

Download

Read: RTECS - Chapter 3

Week-4, Session-1 Class: Services, Synchronization and Shared Resources - Lecture 5 Video ,
Lecture 5 Notes , Updated Notes

Paper: PIP - "Priority inheritance protocols: an approach to real-time synchronization", Sha,

L.; Rajkumar, R.; Lehoczky, J.P.; Computers, IEEE Transactionson, Volume: 39 Issue: 9, Sep 1990.

HW and HW+FW Implementations of RT Services

- ISRs, task canonical structure, and task/service release
- FPGA/ASIC-based State Machines for Offloading
- Microprocessor/Microcontroller Offloading
- DSP Offloading

SW Implementations of RT Services
- Main+ISR Executive
- Cooperative Non-Preemptive Threads
- Callbacks
- Continuation Functions
- Software State-Machines
- Priority Preemptive Run-to-Completion RTOS
- Time-sliced Traditional Best Effort OS

Synchronization and Resource Issues:

- Problems with Blocking (resources other than CPU, e.g. I/O)
- Break up into more threads (better scheduling control)
- Interrupt driven I/O - e.g. Programmable FIFOs
- Model Blocking Time
- Priority inversion (general concept)
- Unbounded priority inversion problem (mutex C.S.)
- Priority inheritence
- Priority ceiling

Other Practical Problems - Solutions:

1) Estimating C (execution jitter) - use WCET
2) Period jitter - transform to highest frequency
3) ISR and context switch overhead - add to WCET
4) Deadlock/Livelock - HW Watch Dog
5) Bad code spinning and wedging - Sanity Monitoring

Case-Study #2: Mars Pathfinder (As presented from multiple perspectives)

Authoritative Mars Pathfinder Account
Mike Jones Overview or What Happened to Mars Pathfinder
Mars Pathfinder -- WRS Story
Mars Pathfinder -- JPL Story
 PROJECT BACKGROUNDER: CONTINUOUS MEDIA (VIDEO), Video-Lecture-Media, PDF
Download

Read: RTECS -- Chapter 4, 5 & 6

Lab-2 DUE
Start Lab-3

Week-5, Session-1 Class: Real-Time Service System Integration and IO

(RT-Linux User-space and Kernel-space Service Threads and Synchronization) - Lecture 6
Video , Lecture 6 Notes , Updated Notes

Service Synchronization, Communication, and IO

- Using Message Queues to Sync Services and for Communication
- Using Binary Semaphores to Sync Services
- What about IO during Service Execution?

Overlapping CPU and IO Cycles - Latency Hiding

- Initial Release Input - Block DMA and FIFO Sensor Data
- Intermediate IO
- Memory-Mapped IO During Service Processing
- Hiding IO latency with Overlap

System Design with RT Services

- Software process: analysis, design, specification, coding,
unit testing, integration, system testing,
delivery, operations

- Implementation characteristics: tasks, message passing,

synchronization, ISRs,
memory-mapped I/O, bus
interface, signals, shared
memory, stack/data/heap
mgt., comm. interfaces

- Specifying the design of real-time software systems

Data flow and Control flow (function and interfaces)
Extended finite state machines and signal block diagrams
MSC (multi-tasking multi-node protocol/comm model)
 - Processes, threads, and tasks, what's the diff in RT-Linux?

Read: RTECS - Chapter 7 and re-read 4.2, 4.3, 4.4 regarding WCET

Lab-3 DUE

Week-6: EXAM-1 on RTECS Chapters 1-7 (2 hours to complete exam on Canvas)

Week-6, Session-1 Class: Scalable Embedded Systems Architectures - Lecture 7 Video , Lecture 7
Notes , Updated Notes

Intro to PCI & USB Architecture and I/O Architectures

- Intro to PCI bus architecture and device interfaces
- PCI Plug and Play Concept
- USB Plug and Play Concept
- Embedded System PCI Form Factors and Standards
CompactPCI, PC/104+ (PCI+ISA), PMC, PCI-X 1.0a/b, PCI-X 2.0, PCI-Express
Read: Universal Serial Bus, (chapters TBD)

- Discussion on I/O Trends (High speed differential serial)

USB
PCI-Express
10G Ethernet (10GEA), which became IEEE P802.3ae
Open Fabric Alliance for 10/20/40/80Gbps Infiniband
Other High-speed Serial: Firewire, RapidIO, SAS/SATA

- Discussion of ASIC Trends

SoC (Yesterday's Board, Today's Chipset, Tomorrow's ASIC)
Core + I/O (PowerPC 8xx, 82xx)
Reconfigurable (Virtex II)
Configurable (Tensilica)
IP Modules (CPU Cores, Mem Ctlrs., Local Bus)
Off-loading (Today's SW is tomorrow's HW)
- The POSIX 1003.1b compliant Embedded system environment (VxWorks)
- The POSIX Application Programmer's Interface
- POSIX overview: message queues, RT signals, and semaphores
- Clocks and timers

Read: RTECS - Chapter 8

Start Lab-4

Week-7, Session-1 Class: Device Drivers and Characterization of Embedded I/O - Lecture 8 Video
, Lecture 8 Notes , Updated Notes

I/O interfaces
- Digital
- Analog (ADC, DAC interfaces)

Microprocessor interface types (word or block)

- Register-based control, status, data
- Higher rate FIFO I/O
- Block-oriented 1st/3rd party DMA tx/rx between I/O interfaces and memory
- Bus burst transfers and block transfers
- system memory map for MMIO devices - DRAM/SDRAM/DDR, BOOTROM, Flash

External interface types

- CPU local bus IO/MMIO
E.g. PCI 2.x, GPIO, DRAM, Flash
- Point-to-point or switched devices
E.g. RS-232, RS-422, PCI-Express
- Network multi-access devices
E.g. Ethernet

Device interfaces -- introduction to drivers

- User space App interface (driver entry point interface to user processes)
- Device interface (interface to devices)
- ring buffers
- blocking/non-blocking
- ioctl "swiss army knife"
- ISRs and signals/semaphores
- scheduled I/O (handle buffering and processing in task)

Read: RTECS - Chapter 9

Week-8, Session-1 Class: Real-Time Performance, Measuring Performance and HW/SW

Debugging Techniques - Lecture 9 Video , Lecture 9 Notes , Updated Notes
Mid-Term Solutions (Flipped Classroom)

Estimating/Measuring Performance Based on CPU Architecture

 IBM Paper on Performance Monitoring and Tuning

- Measuring/Controlling CPU Efficiency

- Trace Ports (e.g. IBM PowerPC 4xx series, Strong Arm)
- Built-in PMU (Performance Monitoring Units) (e.g. Intel Pentium, Xscale)
- External Methods
- Logic Analyzer Memory Traces (Cache Misses, DMA, Un-cached access)
- Memory Port Markers (Writes to Un-cached Memory)

- Profiling Code by Function or Block

- Linux Software in Circuit Methods (e.g. Linux Ftrace, eLinux Ftrace, Kernel Shark, eLinux
Kernel Shark)
- VxWorks RTOS Software in Circuit Methods (e.g. Workbench Code Coverage using SW
In-Circuit, Workbench Profiler)
- Hardware Supported Profiling (e.g. Intel Vtune)
- Cycle-based profiling
- Event-based profiling

- Cache Coherency
- Harvard I-Cache, D-cache Architecture
- Cache Invalidate, Flush, Lock, Pre-fetch

- Measuring/Controlling I/O Efficiency

- Bus Analyzers - e.g. PCI Event Traces
- Logic Analyzer with Support Package

- PCI Bus Performance Tuning

- Bus Grant Arbiters and Priority Schemes, Minimum Burst Length
- PCI-Express Performance Features - PCI Express Tutorial
- Isochronal Virtual Channels
- Split Transaction

Paper: Zen of BDM

Week-8, Session-2 Class: Embedded Volatile and Non-volatile Memory Devices - Lecture 10
Video , Lecture 10 Notes , Updated Notes

Memory Technologies and Hierarchy

- Harvard Arch. I-Cache, D-Cache (Separate instruction and data cache)
- L1 On-Chip, L2/L3 Off-Chip Cache
- SRAM (Fast, but expensive per bit stored)
- DRAM, SDRAM, DDR (Double Data Rate DRAM), Quad Rate SRAM
 ECC (Error Correcting Circuitry) memory
- Hardware/Software Design
- Hamming encoding for EDAC (Error Detection and Correction) - review handout provided
- ECC and maintenance and handling of SBEs and MBEs
- Firmware ECC Req'ts:
1) initialize memory fully,
2) handle SBE interrupts, MBE check-stop, and
3) background scrubbing

Non-Volatile Memory Devices and Systems

- EPROM, EEPROM, NVRAM, and Flash (NAND, NOR)
- Flash File Systems - TrueFFS
- Solid-State-Disk (SSD) - Intel Tech Journal Article, also available on Papers Read in Class
- Advanced Non-Volatile Memory/Storage Devices (PCM, FRAM, MRAM, Race-Track) -
Overview

Read: RTECS - Chapter 10

Lab-4 DUE

Start Extended Lab and preparation of final submission Report and Canvas Digital Video
Submission

Week-9, Session-1 Class: High Availability vs. High Reliability (Same?) - Lecture 11 Video ,
Lecture 11 Notes , Updated Notes

IBM developerWorks Paper on HA and HR

Definition of High Availability (5 9's)

- System is ready to provide service 99.999 % of any given service year
- System can crash, but must reboot with minimal interruption to services and returning to
fully service availability quickly

Definition of High Reliability

- Formal methods and testing to prove reliability is designed in and built in
- SW Path coverage, statement coverage, and condition coverage criteria
- HW Simulation testing with SW drivers
- HW built-in test (Built-in Logic Analyzers, JTAG, Built-in Performance Monitoring
Counters)
- System Safe Modes

Recovery Concepts
- Conservative Safe Mode Manual Recovery (typical of satellite systems)
 - Automatic Recovery (typical of high availability sytems)

Read: RTECS - Chapter 11, 12, 13, 14

Week-10, Session-1, EXAM-2 REVIEW, Extended Lab Q&A and In-Class Work

Week-10: EXAM-2 - RTECS Chapters 8-14 (2.5 hours to complete exam on Canvas)

Finals Week:EXTENDED LAB DUE ON Canvas DROP-BOX

Grades Finalized per Registrar Calendar

Please note that all course materials should be consulted on Canvas; If for some reason you
can't access Canvas, this page is provided as a backup.

Honesty Policies:

Any suspected violations of the Honor Code will be submitted to our Office of Student Conduct
and Conflict Resolution (OSCCR). Students found responsible by our faculty for violating the
cheating policy of the Honor Code will earn an automatic F in the course. Further non-academic
sanctions may be rendered by the OSCCR. We take these issues seriously and have a
responsibility to all students who uphold the Honor Code, and to the highest industry standards
for which we are preparing students. If you have any questions whatsoever regarding what
collaboration is permissible in the course, consult your instructor directly before proceeding.
Sharing of knowledge between students is highly encouraged; however, each student is
expected to independently create and implement their own project files. Students may find
that they are able to leverage firmware designs from books, magazines, the internet, or their
work environments; however, in these cases, students are expected and required to credit the
source of the information clearly and completely. Again, by default, you are expected to turn in
your own original work and cite any and all portions you did not create. All aspects of the Honor
Code apply.