Top tools for continuous integration in embedded software

Continous Integration Intro

This is the second post in a series of posts reviewing continuous integration for embedded software in IoT (take a look at our last post). This post presents a comparison between five continuous integration tools while examining them from the embedded software development perspective.

When examining continuous integration tools for embedded, we chose to focus on the modularity of the tool. Our thinking is that as every embedded software project (mainly in IoT) is different and the developer will need to be able to adapt the tool to support his unique setup. Another consideration is being able to use the tool if the company providing it decided to shut down the project. Embedded software companies need to make sure the tools they are using will last the project lifetime. Industrial IoT for example, it can span to 15–20 years.

Besides these two considerations, we also checked if the tool can support Hardware In the Loop tests. For “on target” tests, node-locked license toolchains and 3rd party integrations support.

We compared the following options: Jenkins, TeamCity, Travis CI, Circle CI and Buildbot. In this comparison, we’ll only add relevant features for embedded software.

Note — a hosted solution is a cloud-based service, managed by the company that created the tool.

Check out our proposal for how to build a continuous integration process for embedded software – here.

Jenkins

An open source, free to use, automation server. Jenkins supports many version control systems. It’s open source, and many industries are using it, so it’s highly likely you’ll have support for the entire project lifespan. It can run locally or remotely so it supports Hardware In the Loop tests and can run compilers with node locked licenses.

Jenkins installation is simple, but as it is self-hosted initial setup time and maintenance take some time. In Jenkins, you have to rely on Plugins, and there are over 1000 of them that can address your integration needs.

To sum it up, Jenkins is suited for embedded software development but requires resources from the company. As it’s an open source framework, it means you can customize everything from the core to plugins to get all the needed flexibility. You can learn more about Jenkins for embedded here.

TeamCity

An automation server by JetBrains that offers out of the box usability. TeamCity provides integration to Eclipse and has a bunch of plugins for 3rd party integrations as well. It supports many version control systems. TeamCity offers full features in its free version and is only limited in build configurations and agents. It can run locally or remotely so it supports Hardware In the Loop tests and can run compilers with node locked licenses.

TeamCity offers more usable interface than Jenkins, reduces the administrative overhead and has a shorter learning curve than Jenkins.

To sum it up, TeamCity is suited for embedded software development. It’s not an open source so you can’t know for sure you’ll have support for the next 15 years. But it’s a mature tool from JetBrains, so the risk is minimal.

Travis CI

An open source hosted service, which is free for open source projects. Start working with Travis CI is very easy, and it is configured using travis.yml files. Getting started is faster than with TeamCity or Jenkins. Travis CI uses VMs for the build process. Travis CI only supports projects that you host on GitHub. With Travis CI you won’t be able to compile with node-locked tools or run Hardware In the Loop tests.

As Travis CI is a hosted service, it’s not the best fit for embedded software projects. If you are using GitHub, compile with GCC and don’t test on a physical device, it can fit your needs.

CircleCI

CircleCI provides both hosted and on-premise service. CircleCI by far reduces maintenance and setup time in comparison to the other tools we’ve evaluated. Configuration is done using a YAML file. CircleCI is using containers to run jobs and let you choose the level of parallelization that suits your needs (require less system resource than Travis CI). CircleCI supports only Git repositories and integrates into GitHub seamlessly.

With version 2.0, CircleCI allows to reproduce environments locally, so that might work for Hardware In the Loop tests and node-locked licenses (running the container with — -net host), but we haven’t checked it.

Although CircleCI is modern and easy to use, it lacks the needed modularity for embedded software development as with Travis CI, but with version 2.0, they introduce some new features that need to be checked.

Buildbot

An open source, free to use, Python-based tool so it can run everywhere Python runs. Buildbot’s configuration files are Python scripts which mean you get flexibility, ease of use and many capabilities. It supports many version control systems. Buildbot’s learning curve is steep but gives a developer a lot of power. Its installation and configuration is not an easy task and documentation can be found here. It can run locally or remotely thus it supports Hardware In the Loop tests and can run node locked toolchains.

Buildbot is suited for embedded software development for the same reasons Jenkins is, but we tend to like Jenkins more due to the abundance of plugins and tools integrations.

Final Thoughts on Continuous Integration Tools for Embedded

Choosing continuous integration tools for embedded is different in many ways from choosing it for web applications development. It depends on your:

  1. Development toolchain licenses.
  2. Supported platforms.
  3. Version control system.
  4. Project lifetime support need.
  5. Where you need to run your tests.

When choosing a modular continuous integration tool, it comes with long configurations and setup time which is something to consider.