OTX Unit Tests

How can we ensure that the test logic actually does what it's supposed to? OTX, as a programming language, must be tested – just like any other code. The test tests the test – it sounds crazy, but it's necessary: ​​because wherever test logic is implemented, verification is also needed!

---

Contents

1. Introduction

2. Fundamentals of software testing

3. What are unit tests?

4. How do you write a unit test?

5. Tool support

6. Conclusion

---

Introduction

To ensure that the test logic programmed in OTX behaves as expected, EMOTIVE developed the UnitTest extension for OTX. This extension itself is part of OTX and extends the core with test procedures and test cases, setup and teardown procedures, as well as new actions and terms. It is also capable of simulating the entire environment, including diagnostic communication ( PduSimulation extension ), for various test scenarios. The extension was developed according to ISO 29119 "Software Testing" and is similar in functionality to the NUnit test framework. Unit tests and integration tests can be described using the UnitTest extension. Through a separate test project in the OTX framework, the tests are clearly separated from the test logic, ensuring that production test logic and tests can never be delivered together.

Fundamentals of software testing

First, a brief introduction to the basics of software testing. There are essentially three types of tests, as shown in the test pyramid in Figure 1 .

1. Unit tests

   Test the smallest possible, standalone component.

2. Integration tests

   Testing the interaction between multiple components in a system, usually based on specific use cases such as flashing, coding or component reading.

3. End-to-end test

   Test the entire system in an environment that is as realistic as possible.

The higher you go in the pyramid, the more expensive it becomes and the longer the tests take.

Figure 1: Types of tests for software verification

What are unit tests?

Unit tests examine the functionality of the smallest units of a software program. The goal is to test each individual component independently and in isolation from other data and external influences after every change, thus ensuring the software's quality. This is also known as regression testing.

What is a good unit test? A good unit test is described by the so-called FIRST principle: Fast - Independent - Repeatable - Self -Validating - Timely . It must therefore be fast and independent, must always produce the same result upon repetition, it must either pass or fail, and ideally, the test should be written before implementation.

There are essentially two testing methods for unit tests: the specification-oriented testing method and the structure-oriented testing method, see Figure 2 .

Figure 2: Test procedures for unit tests

Specification-based testing is also known as black-box testing because it always tests against the specification. Black-box tests are robust against changes, offer good protection against regression, but must always be created manually. White-box testing examines the internal structure of the code. Unlike black-box tests, white-box tests can be generated automatically but are sensitive to changes. In the OTX environment, black-box tests are typically always used.

How do you write a unit test?

This is achieved through methods such as equivalence class formation and boundary value analysis. Equivalence class formation involves defining value range classes for the parameters, for which the same behavior is expected. These classes are called equivalence classes. Then, a representative is found for each identified class, and a test case is written for it. The goal is to achieve high test coverage with as few test cases as possible. Boundary value analysis is essentially equivalence class formation at the boundaries, since errors are not evenly distributed but occur more frequently at the boundaries.

The following OTL code (OTX in ASCII notation) illustrates this using a simplified example of a test procedure with two input and one output parameter. The test procedure verifies the correct execution of an integer division: the quotient (q) is calculated by dividing the dividend (D) by the divisor (d).

[Test]
// Normal division
[TestCase(D = 10, d = 2, expected q = 5)]
// Division with a negative sign
[TestCase(D = -10, d = -2, expected q = 5)]
[TestCase(D = -10, d = 2, expected q = -5)]
// Division with remainder
[TestCase(D = 11, d = 3, expected q = 3)]
// Division at range border
[TestCase(D = 2147483647, d = 2, expected q = 1073741823)]
// Division from zero
[TestCase(D = 0, d = ValueList(1, -1), expected q = 0)]
// Division with zero
[TestCase(D = ValueList(10, -10, 0), d = 0, exception Exception)]
[Parallelizable]
IntegerDivisionTest(in Integer D, in Integer d, out Integer q = 0)
{
   // Can contain arbitrary OTX code, like a ProcedureCall
   Division(D, d, out q);
}


// OTX procedure which will be tested
procedure Division(in Integer D, in Integer d, out Integer q = 0)
{
   // Division as a simple example for test logic to be tested
   q = D / d;
}

To mark the OTX procedure as a test procedure, it is given the attribute [Test] at the beginning. There are seven test cases, called [TestCase] . Each test case contains the input parameters dividend and divisor, as well as the expected result of the division as a quotient. The first example test case simply calculates 10 divided by 5, which should equal 2. The second test case takes the sign into account. It then checks whether the remainder is truncated when dividing 11 by 3. The next test case approaches the range boundary (here, a 32-bit integer), and then division by zero comes into play. The last test case checks the behavior in case of errors during various divisions by zero. Here, a specific exception, the ArithmeticException, is expected. Since the test procedure has no side effects, it can be parallelized during test execution, which is indicated by a corresponding attribute. This can significantly reduce the execution time of the tests, especially when there are a large number of test cases.

Tool support

Figure 3: Open Test Framework with Unit Test Project

The UnitTest extension offers complete integration with the Open Test Framework . Tests can be created graphically or in the OTL editor and executed automatically in the Test Explorer. Each execution generates a clear test log. Using the so-called test context, various environment scenarios can be simulated via OTX mapping files and also executed automatically. Additionally, there is a test log viewer that can display the differences between various test runs.

A test project can be exported as a PUX file. This file contains all the necessary dependencies for running the tests. A PUX file can be executed independently of the development environment using the included console application, for example, automatically in a CI/CD environment.

Conclusion

The Unit Test Extension closes the gap for ensuring the reliability of OTX test logic within the process. It enables the verification of expected behavior of OTX test logic within an automotive manufacturer's process and the creation of test protocols. Complete integration with the Open Test Framework allows for efficient and convenient writing, execution, and change comparison of test cases. Test cases can be exported as PUX files and automatically executed in a CI/CD environment.