Lessons Learned Creating C++ Embedded Systems

From the mapping robots, to the U.S. traffic control system, to Augmented Reality glasses, overcoming hardware limitations

BEVERLY HILLS, CA  (goshrobin.com) 2023/3/16 – When I was a professor teaching Advanced C++ at the Naval Postgraduate School (NPS), I was also faculty robotics advisor. It was my role to guide graduate students, and sometimes write code, to support the NPS robotics program.

SLAM Mapping Robot

NPS research programs include sea and ground robots. The NPS autonomous sea robot Remus has been operated in collaboration with NASA.


NPS REMUS 100 UUV autonomous sea-mapping torpedo


My work was not with the Remus autonomous sea-mapping torpedo, rather with Yamabico, the NPS ground-mapping robot. If you ever wondered what a real Star Wars R2-D2 would look like with the covers off, here it is…

Yamabico ground-mapping autonomous robot

Yamabico ground-mapping autonomous robot

A Simultaneous Localization and Mapping (SLAM) robot, our Unmanned Ground Vehicle (UGV) prototype used sonar, wheel encoders and computer vision to map its surroundings. It does not have GPS or radio communications. When placed in a building, the robot goes up and down hallways mapping the building interior. Wheel encoders counted wheel rotations to calculate dead reckoning and its sonar mapped walls and avoided collisions.

While today anyone can buy a Roomba that does the same thing better, in 1995 this was advanced technology. Roomba evolved from their similar military mapping robot. This technology also evolved into today’s self-driving vehicles, such as Tesla Autopilot.

Safety-Critical Real-Time Traffic Controller

Three companies produce most of the U.S. ground traffic control systems. I worked for one of them on Smart Cities and AI systems integration and code refactoring.

Econolite Cobalt traffic light controller

Econolite Cobalt traffic light controller


Typical next to U.S. intersections is a nondescript silver utility box. Inside that box is a traffic controller, an autonomous embedded system that controls when the light changes and the crosswalk warnings.

To prevent the disaster of a traffic light ever showing green in all directions at the same time, the embedded system has second processor on a watchdog timer. If the controller starts to behave unexpectedly, the watchdog will shut it down within a tenth of second, putting the light into flash mode. That is, blinking red in all directions. A security measure, the controller is autonomous. It can only be advised by a city central traffic controller. The embedded system will reject any orders received it considers unsafe.

AI for Smart Cities is coming. This greatly enhanced software is to run on legacy embedded traffic controllers that have a 10-year-expected lifespan. To do that, the traffic controller needed significant optimization. Must run software twice as fast on the same hardware. Otherwise, adding AI features would overload the PowerPC processor, trigger a watchdog timeout fail-safe event. When a traffic light is blinking red in all directions, that’s the system fail-safe mode.

For added safety, a truck-roll is required to reset a traffic controller. It must be rebooted in person. Is impossible to hack a traffic light with a remote DOS attack. At worst, the watchdog will take over and put it into flash. By the way, experimental traffic light controllers on a university test track may not have these safety features, and have been successfully hacked.

My approach to speed up the system involved removing system and thread waits. That made everything much faster indeed, but with resultant race conditions that I had to fix without using waits. That could mean setting a flag and timer to tell the system that a piece of hardware is present, but in power-up and unstable, to ignore it temporarily. Any crazy data not handled would confuse the controller, trigger the dreaded watchdog. The old way, using waits to give hardware time to settle down had worked, but was too expensive.

The optimization changes made the system faster and more robust. I also did debugging with remote gdb to fix legacy bugs that lurked in the system. Some latent bugs became apparent as the system sped up.

Satellite Set-top Box Security and 500-Channel Expansion

Satellite delivers television everywhere in America. Satellite set-top box (STB) security is important, to authenticate users who’ve paid to access the service, and to keep out pirates.

AT&T DIRECTV Genie Set-top Box

AT&T DIRECTV Genie Set-top Box

Some say writing 50-page engineering specifications using Microsoft Word is boring work, but I found it exciting. For each new STB feature I received a half-page business specification from the CTO, defining what results are expected, how the user would see the new feature. Then it was my job to define how to make it so, to write implementation requirements for engineers.

A second-room remote receiver was being added to the STB. The main and secondary devices boot up separately and communicate with each other. What I must make sure does not happen is deadlock, that both devices somehow get into a state waiting for the other to say it is safe to proceed.

In addition to the secondary STB, the system was getting a satellite upgrade. A new satellite would be launched into geosynchronous orbit nearby the existing satellite. Existing satellite dishes would not need to be re-aimed to see the new satellite. That meant both satellites would be received on the same dish. To avoid signal interference, we must not conflict in their frequencies. Both satellites would be in operation until eventually the old satellite is shut down.

All of this new functionality must be compatible with legacy set-top boxes, not only the new top-of-the-line model that I was tasked to write the specs. No pressure. If I made a mistake, I could mess up every STB in America. Because the specs are written before the design is approved and the team assembled, I recruited my own ad hoc engineering team. I went around the building asking who had worked on something like this before, called us together for a meeting, and got feedback on my design specification.

Action Sports Camera IoT and GUI

A popular sports camera manufacturer wanted to connect everything together using IoT and at the same time to redesign all their apps to use the same programming language and GUI API.

GoPro Hero 4 sports action camera popular with surfers and skiers

GoPro Hero 4 sports action camera popular with surfers and skiers

For IoT, I used MQTT Mosquitto. For GUI, I developed with Qt. I wrote some software twice, using node.js and JavaScript to create a Web version. Sometimes, I fixed bugs in our code on our live system on the AWS cloud, that it was urgent to fix a bug users encountered in our live system. In an emergency, I could be pulled off of R&D to fix code I’d never seen before.

Augmented Reality Glasses

Magic Leap and Microsoft HoloLens, pioneers in the AR glasses market, were exciting products, but expensive. Could we build AR glasses with similar features, but better priced for mass market?

Lenovo ThinkReality Augmented Reality Glasses

Lenovo ThinkReality Augmented Reality Glasses


Lenovo ThinkReality Smart Glasses featured a Qualcomm Snapdragon 845 processor, accelerometer, ambient light sensor, gyro, depth sensor, camera, Bluetooth, WiFi and NFC. AR glasses have no keyboard or mouse, rely heavily on computer vision and speech. My task was to prototype the object recognition and speech recognition software. I also developed a cloud data API to store information to-and-from the glasses. And, I did R&D in hands-free programming interfaces.

When engineering any wearable device strapped to someone’s body, concerns are heat and power. I was rightly concerned that my AI software running on the glasses could consume too mluch battery power and create too much heat in the process. If that’s allowed to get too happen, the unit could go into safe-mode, a low power setting to allow the glasses to cool down.

Without good battery power management, batteries might be drained within 15 minutes. My suggestion for battery challenges included hardware, to add a circuit to allow hot-swapping batteries. That would avoid a reboot, but the company decided it was too late in manufacturing to make even a minor hardware change. I’d have to work with the hardware we had.

Lessons Learned

For safety, pilots are drilled to think about altitude and speed, to not get into a spin or collide with the ground. And also fuel, to not run out and fall to Earth. Embedded systems development is similar, where heat is like altitude, processor load is speed, and battery power is fuel.

About Robin Rowe

Robin Rowe

Robin Rowe, has designed and programmed embedded systems, is a Hollywood creative technologist, engineering director, product designer, AI research lab director, and C++ software architect. Robin has developed consumer, enterprise and safety-critical products including Lenovo ThinkReality Augmented Reality glasses, AT&T DIRECTV Genie satellite set-top box, Econolite Cobalt Linux embedded system DOT traffic controller and Apple Appstore and Google Playstore mobile games.