coreboot-kgpe-d16/Documentation/tutorial/part3.md

14 KiB

Writing unit tests for coreboot

Introduction

General thoughts about unit testing coreboot can be found in Unit-testing coreboot. Additionally, code coverage support is available for unit tests.

This document aims to guide developers through the process of adding and writing unit tests for coreboot modules.

As an example of unit-under-test, src/device/i2c.c (referred hereafter as UUT "Unit Under Test") will be used. This is simple module, thus it should be easy for the reader to focus solely on the testing logic, without the need to spend too much time on digging deeply into the source code details and flow of operations. That being said, a good understanding of what the unit-under-test is doing is crucial for writing unit tests.

This tutorial should also be helpful for developers who want to follow TDD. Even though TDD has a different work flow of building tests first, followed by the code that satisfies them, the process of writing tests and adding them to the tree is the same.

Analysis of unit-under-test

First of all, it is necessary to precisely establish what we want to test in a particular module. Usually this will be an externally exposed API, which can be used by other modules.

.. admonition:: i2c-test example

   In case of our UUT, API consist of two methods:

   .. code-block:: c

     int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg,
                 uint8_t *data, uint8_t mask, uint8_t shift)
     int i2c_write_field(unsigned int bus, uint8_t chip, uint8_t reg,
                 uint8_t data, uint8_t mask, uint8_t shift)

   For sake of simplicity, let's focus on `i2c_read_field` in this
   document.

Once the API is defined, the next question is what this API is doing (or what it will be doing in case of TDD). In other words, what outputs we are expecting from particular functions, when providing particular input parameters.

.. admonition:: i2c-test example

   .. code-block:: c

     int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg,
                 uint8_t *data, uint8_t mask, uint8_t shift)

   This is a method which means to read content of register `reg` from
   i2c device on i2c `bus` and slave address `chip`, applying bit `mask`
   and offset `shift` to it. Returned data should be placed in `data`.

The next step is to determine all external dependencies of UUT in order to mock them out. Usually we want to isolate the UUT as much as possible, so that the test result depends only on the behavior of UUT and not on the other modules. While some software dependencies may be hard to be mock (for example due to complicated dependencies) and thus should be simply linked into the test binaries, all hardware dependencies need to be mocked out, since in the user-space host environment, target hardware is not available.

.. admonition:: i2c-test example

   `i2c_read_field` is calling `i2c_readb`, which eventually invokes
   `i2c_transfer`. This method simply calls `platform_i2c_transfer`. The
   last function in the chain is a hardware-touching one, and defined
   separately for different SOCs. It is responsible for issuing
   transactions on the i2c bus.  For the purpose of writing unit test,
   we should mock this function.

Adding new tests

In order to keep the tree clean, the tests/ directory should mimic the src/ directory, so that test harness code is placed in a location corresponding to UUT. Furthermore, the naming convention is to add the suffix -test to the UUT name when creating a new test harness file.

.. admonition:: i2c-test example

   Considering that UUT is `src/device/i2c.c`, test file should be named
   `tests/device/i2c-test.c`. When adding a new test file, it needs to
   be registered with the coreboot unit testing infrastructure.

Every directory under tests/ should contain a Makefile.inc, similar to what can be seen under the src/. Register a new test in Makefile.inc, by appending test name to the tests-y variable.

.. admonition:: i2c-test example

   .. code-block:: c

     tests-y += i2c-test

Next step is to list all source files, which should be linked together in order to create test binary. Usually a tests requires only two files

  • UUT and test harness code, but sometimes more is needed to provide the test environment. Source files are registered in <test_name>-srcs variable.
.. admonition:: i2c-test example

   .. code-block:: c

     i2c-test-srcs += tests/device/i2c-test.c
     i2c-test-srcs += src/device/i2c.c

Above minimal configuration is a basis for further work. One can try to build and run test binary either by invoking make tests/<test_dir>/<test_name> or by running all unit tests (whole suite) for coreboot make unit-tests.

.. admonition:: i2c-test example

   .. code-block:: c

     make tests/device/i2c-test

   or

   .. code-block:: c

     make unit-tests

When trying to build test binary, one can often see the linker complaining about undefined reference for a couple of symbols. This is one of the solutions to determine all external dependencies of UUT - iteratively build test and resolve errors one by one. At this step, developer should decide either it's better to add an extra module to provide necessary definitions or rather mock such dependency. A quick guide about adding mocks is provided later in this doc.

Writing new tests

In coreboot, Cmocka is used as unit test framework. The project has exhaustive API documentation. Let's see how we may incorporate it when writing tests.

Assertions

Testing the UUT consists of calling the functions in the UUT and comparing the returned values to the expected values. Cmocka implements a set of assert macros to compare a value with an expected value. If the two values do not match, the test fails with an error message.

.. admonition:: i2c-test example

   In our example, the simplest test is to call UUT for reading our fake
   devices registers and do all calculation in the test harness itself.
   At the end, let's compare integers with `assert_int_equal`.

   .. code-block:: c

     #define MASK        0x3
     #define SHIFT        0x1

     static void i2c_read_field_test(void **state)
     {
             int bus, slave, reg;
             int i, j;
             uint8_t buf;

             mock_expect_params_platform_i2c_transfer();

             /* Read particular bits in all registers in all devices, then compare
                with expected value. */
             for (i = 0; i < ARRAY_SIZE(i2c_ex_devs); i++)
                     for (j = 0; j < ARRAY_SIZE(i2c_ex_devs[0].regs); j++) {
                             i2c_read_field(i2c_ex_devs[i].bus,
                                     i2c_ex_devs[i].slave,
                                     i2c_ex_devs[i].regs[j].reg,
                                     &buf, MASK, SHIFT);
                             assert_int_equal((i2c_ex_devs[i].regs[j].data &
                                     (MASK << SHIFT)) >> SHIFT, buf);
                     };
     }

Mocks

Overview

Many coreboot modules are low level software that touch hardware directly. Because of this, one of the most important and challenging part of writing tests is to design and implement mocks. A mock is a software component which implements the API of another component so that the test can verify that certain functions are called (or not called), verify the parameters passed to those functions, and specify the return values from those functions. Mocks are especially useful when the API to be implemented is one that accesses hardware components.

When writing a mock, the developer implements the same API as the module being mocked. Such a mock may, for example, register a set of driver methods. Behind this API, there is usually a simulation of real hardware.

.. admonition:: i2c-test example

   For purpose of our i2c test, we may introduce two i2c devices with
   set of registers, which simply are structs in memory.

   .. code-block:: c

     /* Simulate two i2c devices, both on bus 0, each with three uint8_t regs
        implemented. */
     typedef struct {
             uint8_t reg;
             uint8_t data;
     } i2c_ex_regs_t;

     typedef struct {
             unsigned int bus;
             uint8_t slave;
             i2c_ex_regs_t regs[3];
     } i2c_ex_devs_t;

     i2c_ex_devs_t i2c_ex_devs[] = {
             {.bus = 0, .slave = 0xA, .regs = {
                     {.reg = 0x0, .data = 0xB},
                     {.reg = 0x1, .data = 0x6},
                     {.reg = 0x2, .data = 0xF},
             } },
             {.bus = 0, .slave = 0x3, .regs = {
                     {.reg = 0x0, .data = 0xDE},
                     {.reg = 0x1, .data = 0xAD},
                     {.reg = 0x2, .data = 0xBE},
             } },
     };

   These fake devices will be accessed instead of hardware ones:

   .. code-block:: c

             reg = tmp->buf[0];

             /* Find object for requested device */
             for (i = 0; i < ARRAY_SIZE(i2c_ex_devs); i++, i2c_dev++)
                     if (i2c_ex_devs[i].slave == tmp->slave) {
                             i2c_dev = &i2c_ex_devs[i];
                             break;
                     }

             if (i2c_dev == NULL)
                     return -1;

             /* Write commands */
             if (tmp->len > 1) {
                     i2c_dev->regs[reg].data = tmp->buf[1];
             };

             /* Read commands */
             for (i = 0; i < count; i++, tmp++)
                     if (tmp->flags & I2C_M_RD) {
                             *(tmp->buf) = i2c_dev->regs[reg].data;
                     };

Cmocka uses a feature that gcc provides for breaking dependencies at the link time. It is possible to override implementation of some function, with the method from test harness. This allows test harness to take control of execution from binary (during the execution of test), and stimulate UUT as required without changing the source code.

coreboot unit test infrastructure supports overriding of functions at link time. This is as simple as adding a name_of_function to be mocked into <test_name>-mocks variable in Makefile.inc. The result is that the test's implementation of that function is called instead of coreboot's.

.. admonition:: i2c-test example

   .. code-block:: c

     i2c-test-mocks += platform_i2c_transfer

   Now, dev can write own implementation of `platform_i2c_transfer`.
   This implementation instead of accessing real i2c bus, will
   write/read from fake structs.

   .. code-block:: c

     int platform_i2c_transfer(unsigned int bus, struct i2c_msg
                 *segments, int count)
     {
     }

Checking mock's arguments

A test can verify the parameters provided by the UUT to the mock function. The developer may also verify that number of calls to mock is correct and the order of calls to particular mocks is as expected (See this). The Cmocka macros for checking parameters are described here. In general, in mock function, one makes a call to check_expected(<param_name>) and in the corresponding test function, expect*() macro, with description which parameter in which mock should have particular value, or be inside a described range.

.. admonition:: i2c-test example

   In our example, we may want to check that `platform_i2c_transfer` is
   fed with a number of segments bigger than 0, each segment has flags
   which are in the supported range and each segment has a buf which is
   non-NULL. We are expecting such values for _every_ call, thus the
   last parameter in `expect*` macros is -1.

   .. code-block:: c

     static void mock_expect_params_platform_i2c_transfer(void)
     {
             unsigned long int expected_flags[] = {0, I2C_M_RD,
                     I2C_M_TEN, I2C_M_RECV_LEN, I2C_M_NOSTART};

             /* Flags should always be only within supported range */
             expect_in_set_count(platform_i2c_transfer, segments->flags,
                     expected_flags, -1);

             expect_not_value_count(platform_i2c_transfer, segments->buf,
                     NULL, -1);

             expect_in_range_count(platform_i2c_transfer, count, 1,
                     INT_MAX, -1);
     }

   And the checks below should be added to our mock

   .. code-block:: c

             check_expected(count);

             for (i = 0; i < count; i++, segments++) {
                     check_expected_ptr(segments->buf);
                     check_expected(segments->flags);
             }

Instrument mocks

It is possible for the test function to instrument what the mock will return to the UUT. This can be done by using the will_return*() and mock() macros. These are described in the Mock Object section of the Cmocka API documentation.

.. admonition:: Example

   There is an non-coreboot example for using Cmocka available
   `here <https://lwn.net/Articles/558106/>`_.

Test runner

Finally, the developer needs to implement the test main() function. All tests should be registered there and the cmocka test runner invoked. All methods for invoking Cmocka test are described here.

.. admonition:: i2c-test example

   We don't need any extra setup and teardown functions for i2c-test, so
   let's simply register the test for `i2c_read_field` and return from
   main the output of Cmocka's runner (it returns number of tests
   that failed).

   .. code-block:: c

     int main(void)
     {
             const struct CMUnitTest tests[] = {
                     cmocka_unit_test(i2c_read_field_test),
             };

             return cb_run_group_tests(tests, NULL, NULL);
     }