IoT devices are open to traditional control flow integrity (CFI) attacks resulting from buffer overflow and return-oriented programming like techniques. They often have limited computational capacity ruling out many of the traditional heavy-duty software countermeasures. In this work, we deploy hardware/software solutions to detect CFI attacks. Some of the medium capability IoT devices, for example based on Raspberry Pi, contain ARM Cortex A-53 (Pi 3) or Cortex A-73 (Pi 4) processors. These processors include hardware counters to count microarchitecture level events affecting performance. Lighter weight IoT devices, say based on ARM Cortex M4 or M7, include DWT (Debug, Watch & Trace) module. When control flow anomalies caused by attacks such as buffer overflow or return oriented programming (ROP) occur, they leave a microarchitectural footprint. Hardware counters reflect such footprints to flag control flow anomalies. This paper is geared towards buffer overflow and ROP control flow anomaly detection in embedded programs. The targeted program entities are main event loops and task/event handlers. The proposed anomaly detection mechanism is evaluated on ArduPilot [1] - a popular autopilot software on a Raspberry Pi 3 with PMU and DWT. A self-navigation program is evaluated on an iCreate Roomba platform with an ARM Cortex M4 processor with DWT only. We are able to achieve 97-99%+ accuracy with 1-10 micro-second time overhead per control flow anomaly check.