Automated Testing in Embedded Software – Why?

Automated testing in embedded software has many benefits. When implemented in your organization it can:

  1. Reduce the number of bugs discovered late in the development cycle.
  2. Help you refactor your code with confidence.
  3. Reduce recovery time for post-production failures.
  4. Improve time to market and build a more productive R&D cycle.
  5. Increase features delivery and drives business initiatives.

Yes, it sounds promising, and it is! (check out the transformation HP made)

Moving from manual to automated testing in embedded software

In this blog post, we’ll discuss the five steps you must go through to move from manual to automated testing in embedded software. I’ll try to help you make it fast and easy.

Let’s start then.

1. Internal evangelism

As with any transformation, this transformation starts with convincing the decision makers, stakeholders, and developers (they must be your allies). From my experience, this is the hardest and the most significant task.

People are accustomed to their old methodologies and beliefs.

From the decision maker and stakeholders, you’ll be hearing a lot of:

  1. What’s the ROI of test automation?
  2. Do you want me to stop developing features and build a testing infrastructure?
  3. Let’s start on the next project.
  4. We must focus all our effort on the current version.
  5. I don’t have the personnel to start a test automation initiative.
  6. Why fix it if it’s not broken?

From developers and test engineers, you’ll get a lot of:

  1. Automating tests on the physical hardware is hard.
  2. I don’t have enough testing equipment for the job.
  3. How am I going to trigger physical inputs of an embedded device?
  4. We have a noisy environment; I can’t rely on the output as it’s not consistent.
  5. There’s no proper testing infrastructure for embedded systems.
  6. Hardware is flaky and noisy – how are you going to automate on non-stable hardware?

In my last position, I was the CEO of an embedded systems startup company. I was focused on features delivery and suffered from production bugs all the time. But only when a production bug almost shut down our business, I understood that test automation is a must-have – no matter the initial investment.

If you’d like to convince the decision makers in your organization, show them some devastating outcomes for software bugs, studies on the ROI of test automation and continuous integration and successful case studies, or tell them to contact me! And… don’t forget to learn about tools and frameworks for automated testing in embedded software that are specific to your domain. When pitching to decision makers, you must come prepared.

To handle the R&D team concerns some of the essentials are in the next steps. Stay tuned to our next posts for more information.

2. Automatic builds

Every big revolution starts with a first step. In my opinion, it must be an easy one. The first step for test automation is automated builds. Here you need to have a script that upon a specific Git event will issue a build task for your code and will report the result. This simple step will let you know if your changed code can compile and produce an executable. This executable will also become handy on the next steps of your test automation process.

When you show your peers and decision makers a report of all the automated builds, they’ll start seeing the bigger picture. Use the posts below to learn how to set up automated build process:

  1. Four tools for automated build in embedded software.
  2. nRF52 build server – how to setup?
  3. STM32 build server – how to setup?

3. Small and automated test cases

The next step is building three types of tests. Note that these tests must be small, have reproducible inputs and consistent results. Remember, in this phase, we aim for small successes with minimum risk of failure. The test types will be:

  1. Unit test – choose one unit, mock the interfaces, automate the test execution.
  2. Black box test (for your embedded device) – choose a flow you want to cover, trigger the inputs, monitor the outputs, automated the test execution.
  3. System test (embedded device with the backend and other components) – choose one flow you want to cover from your backend to your device, trigger a scenario, record the outcome, automate the test execution.

This is the first tricky step, it will require the building of a small testing infrastructure but it worth your time!

If you are using Segger tools and want to learn how to use it for test automation in embedded software, read here.

4. Build an automated report system

You now have an automated build and three test cases that run automatically upon specific Git events. The next step will be to build a reporting system. This reporting system can be an email being sent after each run with the status of each step – again start with something small that doesn’t require much effort. As I already noted think fast, think easy.

With a reporting system in place, you, your team and the decision makers can get a visual notification of how each code change is affecting the quality of the system.

5. Zero tolerance to a failing test

This step is more of a mindset change. To start with test automation you and your organization must get used to fixing the code or the test when a test is failing. I can’t emphasize enough on how important this is – this is something you must practice. False positives and flaky tests are the main reason your team members will reject the use of automation. If you start cutting corners here, your entire automated test initiative will fall.

Let’s Recap

In this blog post, I’ve covered the five must-have steps towards an organizational change from manual to automated testing in embedded software. I’ve not covered the tools yet, but I did cover practical methods you can start with. The goal here is to show your organization a better way of testing embedded systems. To learn about the tools, stay tuned – it will be covered in my next posts.

On a personal note, back in in the time when I lead an embedded systems company, I wish that someone from my R&D team would have pushed the initiative of test automation.

Now it’s up to you to drive this change (from there you can move to continuous integration)!