Continuous integration for embedded software

Continous Integration for Embedded Software – Why?

Researches of SW R&D teams show significant improvements in all R&D and operation metrics when proper continuous integration and delivery processes were implemented (take a look at this case study). We thought it would be highly beneficial to demonstrate how to apply these practices to embedded SW teams. Applying continuous integration for embedded software development is not an easy task as you have a physical device with many constraints when it comes to rapid testing. However, the gained benefits outweigh the effort. For more info about CI advantages and considerations when it comes to embedded SW check out this post. For a comparison of different tools for continuous integration read this post.

In this post, I’ll outline a guideline for building continuous integration for embedded software development. Note that this post is relevant even if your company privacy policy implies restrictions on using cloud services for source control and other CI/CD tools. This post will be focused on the IT infrastructure and tools you need in order to implement CI/CD process in your organization.

Continous integration for embedded software flow overview

The basic continuous integration for embedded software flow will be one of this kind

continuous integration for embedded software flow

Let’s dig down further and discuss the details of our suggested flow:

continuous integration for embedded software elaborated flow

We chose to work with the Git Flow proposed by Vincent Driessen. In a future post, we’ll dig down further and discuss how to implement Git Flow for embedded SW.

Update: here’s the Git Flow post link.

To enable this, we propose the following IT infrastructure.

IT Infrastructure for continuous integration for embedded software 

continuous integration for embedded software setup overview

When building an IT infrastructure for continuous integration for embedded software we decided to use more flexible tools that can work on-premise and as hosted cloud services (there are companies that have strict privacy and security policies which prevent them from using hosted cloud solutions). Additionally, toolchains such as IAR that have a node lock license, and when doing on target tests, you must have an on-prem server to serve as your test harness. All of these components run on VMs (either windows or Linux ones). We decided to use different VMs for different types of test for modularity and support for future infrastructure updates.

Continuous Integration Master Server

We decided to use Jenkins as our CI/CD manager. Jenkins can run on-prem and on the cloud, on Windows/Linux machined, supports HiL tests and everything you need for embedded SW development. We are also very familiar with it, so it was a straightforward choice for us to make.

Source Control Server

We decided to use GitLab as it supports both on-prem and hosted cloud service. GitLab has a very friendly user interface and reliable plugin to Jenkins. In an on-prem scenario, we also connected the VM to a central backup service. If you don’t have an on-prem constraint, GitHub is probably the way to go.

Continuous Integration for Embedded Software Build Server

Our build server will also perform static analysis tests. For a build server, we used one of the two options:

  1. IAR build server — an on-prem Windows VM with Node lock license. Note that static analysis tools require a more expensive IAR license.
  2. GCC ARM build server — can be on-prem or cloud-based Linux VM (or even Docker-based). Check out these posts about automated STM32 build server and automated nRF52 build server to learn more.

Non-Host Testing Server

A Linux based VM, where you can run all non-target tests — unit tests and module testing with mocking. Our favorite tools for unit tests for embedded C are the tools provided by ThrowTheSwich.org — Unity, CMock, and Ceedling.

Simulated Testing Server

A Linux based VM, where you can run your binaries on a simulator of your devices and your entire system. There are few options for that like Synopsys’ VDK, WindRiver’s Simics, QEMU, Mentor’s Vista, etc. We recommend our own solution — Jumper’s Virtual Lab (we’ll have a different post covering this as well). A proper simulation/emulation environment should allow you to run-

  1. Unit tests
  2. Simulating sensors and peripherals I/Os
  3. Smoke tests — device and cluster of devices
  4. Black box (end to end) tests
  5. System level tests — load/stress, performance, security, etc. tests
  6. Cluster (scalability) testing with communication simulation
  7. Acceptance tests
  8. Performance tests

Check this blog post, so see a working continuous integration flow with build and test automation on STM32.

Target HW Testing Server

A Linux/Windows VM (the physical server should be located in the physical lab), where you can run tests on the target HW. This server will act as the testing harness or will be connected to an external HW testing harness. This server will control lab testing equipment such as scope, network analyzer, voltmeter, logic, packet generator, etc. We haven’t chosen specific testing framework here, as it depends on the specific device and system setup. On the target device and physical lab you should run –

  1. Smoke tests
  2. Black box tests
  3. System level tests
  4. HW validation and timing related tests
  5. Cluster testing with communication testing (note that the scalability here is limited)
  6. Certification tests
  7. Acceptance tests

Streamline Your Continous Integration Flow and Automated Testing

The fundamental requirement for continuous integration flow is to have a fast feedback loop from code commit through integration to test results. The way to get there is through automating your process and mainly having automated tests. At Jumper, we’ve built a Virtual Lab platform to streamline test automation and continuous integration for embedded software. Jumper’s Virtual Lab allows R&D and QA teams to have a virtual production clone that runs the same physical device’s software executables with complete environment and communication simulation.


If you work with embedded systems and want to have an easy way of testing your physical device’s embedded software, you’re invited to try Jumper.

Summary

Building an infrastructure for continuous integration for embedded software in your organization will require some effort from your R&D and DevOps/IT teams. It will require new budget allocation to buy tools to support this process, and you’ll probably need to implement new development practices with your R&D team. Nevertheless, when it will be fully embedded into your R&D cycle you’ll get significant ROI very soon. Read here on how to move from manual testing to automated testing in embedded, it might give you some ideas.

Share Your Thoughts

Feel free to share your thoughts and contact us in the comments section or on feedback@jumper.io.