Documentation/tutorial: Add tutorial for writing unit tests
Signed-off-by: Jan Dabros <jsd@semihalf.com> Change-Id: I1ebd2786a49ec8bc25e209d67ecc4c94b475442d Reviewed-on: https://review.coreboot.org/c/coreboot/+/41727 Tested-by: build bot (Jenkins) <no-reply@coreboot.org> Reviewed-by: Paul Fagerburg <pfagerburg@chromium.org> Reviewed-by: Patrick Georgi <pgeorgi@google.com>
This commit is contained in:
parent
23e8b5b494
commit
5694342a81
|
@ -279,41 +279,6 @@ tests/lib/string-test and tests/device/i2c-test:
|
|||
├── i2c.o
|
||||
```
|
||||
|
||||
### Adding new tests
|
||||
For purpose of this description, let's assume that we want to add a new unit test
|
||||
for src/device/i2c.c module. Since this module is rather simple, it will be enough
|
||||
to have only one test module.
|
||||
|
||||
Firstly (assuming there is no tests/device/Makefile.inc file) we need to create
|
||||
Makefile.inc in main unit test module directory. Inside this Makefile.inc, one
|
||||
need to register new test and can specify multiple different attributes for it.
|
||||
|
||||
```bash
|
||||
# Register new test, by adding its name to tests variable
|
||||
tests-y += i2c-test
|
||||
|
||||
# All attributes are defined by <test_name>-<attribute> variables
|
||||
# <test_name>-srcs is used to register all input files (test harness, unit under
|
||||
# test and others) for this particular test. Remember to add relative paths.
|
||||
i2c-test-srcs += tests/device/i2c-test.c
|
||||
i2c-test-srcs += src/device/i2c.c
|
||||
|
||||
# We can define extra cflags for this particular test
|
||||
i2c-test-cflags += -DSOME_DEFINE=1
|
||||
|
||||
# For mocking out external dependencies (functions which cannot be resolved by
|
||||
# linker), it is possible to register a mock function. To register new mock, it
|
||||
# is enough to add function-to-be-mocked name to <test_name>-mocks variable.
|
||||
i2c-test-mocks += platform_i2c_transfer
|
||||
|
||||
# Similar to coreboot concept, unit tests also runs in the context of stages.
|
||||
# By default all unit tests are compiled to be ramstage executables. If one want
|
||||
# to overwrite this setting, there is <test_name>-stage variable available.
|
||||
i2c-test-stage:= bootblock
|
||||
```
|
||||
|
||||
### Writing new tests
|
||||
Full description of how to write unit tests and Cmocka API description is out of
|
||||
the scope of this document. There are other documents related to this
|
||||
[Cmocka API](https://api.cmocka.org/) and
|
||||
[Mocks](https://lwn.net/Articles/558106/).
|
||||
Our tutorial series has [detailed guidelines](../tutorial/part3.md) for writing
|
||||
unit tests.
|
||||
|
|
|
@ -2,3 +2,4 @@
|
|||
|
||||
* [Part 1: Starting from scratch](part1.md)
|
||||
* [Part 2: Submitting a patch to coreboot.org](part2.md)
|
||||
* [Part 3: Writing unit tests](part3.md)
|
||||
|
|
|
@ -0,0 +1,384 @@
|
|||
# Writing unit tests for coreboot
|
||||
|
||||
## Introduction
|
||||
General thoughts about unit testing coreboot can be found in
|
||||
[Unit testing coreboot](../technotes/2020-03-unit-testing-coreboot.md).
|
||||
|
||||
This document aims to guide developers through the process of adding and writing
|
||||
unit tests for coreboot modules.
|
||||
|
||||
As an example of unit under test, `src/device/i2c.c` (referred hereafter as UUT
|
||||
"Unit Under Test") will be used. This is simple module, thus it should be easy
|
||||
for the reader to focus solely on the testing logic, without the need to spend
|
||||
too much time on digging deeply into the source code details and flow of
|
||||
operations. That being said, a good understanding of what the unit under test is
|
||||
doing is crucial for writing unit tests.
|
||||
|
||||
This tutorial should also be helpful for developers who want to follow
|
||||
[TDD](https://en.wikipedia.org/wiki/Test-driven_development). Even though TDD
|
||||
has a different work flow of building tests first, followed by the code that
|
||||
satisfies them, the process of writing tests and adding them to the tree is the
|
||||
same.
|
||||
|
||||
## Analysis of unit under test
|
||||
First of all, it is necessary to precisely establish what we want to test in a
|
||||
particular module. Usually this will be an externally exposed API, which can be
|
||||
used by other modules.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
In case of our UUT, API consist of two methods:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t *data,
|
||||
uint8_t mask, uint8_t shift)
|
||||
int i2c_write_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t data,
|
||||
uint8_t mask, uint8_t shift)
|
||||
|
||||
For sake of simplicity, let's focus on `i2c_read_field` in this document.
|
||||
```
|
||||
|
||||
Once the API is defined, the next question is __what__ this API is doing (or
|
||||
what it will be doing in case of TDD). In other words, what outputs we are
|
||||
expecting from particular functions, when providing particular input parameters.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t *data,
|
||||
uint8_t mask, uint8_t shift)
|
||||
|
||||
This is a method which means to read content of register `reg` from i2c device
|
||||
on i2c `bus` and slave address `chip`, applying bit `mask` and offset `shift`
|
||||
to it. Returned data should be placed in `data`.
|
||||
```
|
||||
|
||||
The next step is to determine all external dependencies of UUT in order to mock
|
||||
them out. Usually we want to isolate the UUT as much as possible, so that the
|
||||
test result depends __only__ on the behavior of UUT and not on the other
|
||||
modules. While some software dependencies may be hard to be mock (for example
|
||||
due to complicated dependencies) and thus should be simply linked into the test
|
||||
binaries, all hardware dependencies need to be mocked out, since in the
|
||||
user-space host environment, targets hardware is not available.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
`i2c_read_field` is calling `i2c_readb`, which eventually invokes
|
||||
`i2c_transfer`. This method simply calls `platform_i2c_transfer`. The last
|
||||
function in the chain is a hardware-touching one, and defined separately for
|
||||
different SOCs. It is responsible for issuing transactions on the i2c bus.
|
||||
For the purpose of writing unit test, we should mock this function.
|
||||
```
|
||||
|
||||
## Adding new tests
|
||||
In order to keep the tree clean, the `tests/` directory should mimic the `src/`
|
||||
directory, so that test harness code is placed in a location corresponding to
|
||||
UUT. Furthermore, the naming convention is to add the suffix `-test` to the UUT
|
||||
name when creating a new test harness file.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
Considering that UUT is `src/device/i2c.c`, test file should be named
|
||||
`tests/device/i2c-test.c`. When adding a new test file, it needs to be
|
||||
registered with the coreboot unit testing infrastructure.
|
||||
```
|
||||
|
||||
Every directory under `tests/` should contain a Makefile.inc, similar to what
|
||||
can be seen under the `src/`. Register a new test in Makefile.inc, by
|
||||
__appending__ test name to the `tests-y` variable.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
tests-y += i2c-test
|
||||
```
|
||||
|
||||
Next step is to list all source files, which should be linked together in order
|
||||
to create test binary. Usually a tests requires only two files - UUT and test
|
||||
harness code, but sometimes more is needed to provide the test environment.
|
||||
Source files are registered in `<test_name>-srcs` variable.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
i2c-test-srcs += tests/device/i2c-test.c
|
||||
i2c-test-srcs += src/device/i2c.c
|
||||
```
|
||||
|
||||
Above minimal configuration is a basis for further work. One can try to build
|
||||
and run test binary either by invoking `make tests/<test_dir>/<test_name>` or by
|
||||
running all unit tests (whole suite) for coreboot `make unit-tests`.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
make tests/device/i2c-test
|
||||
|
||||
or
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
make unit-tests
|
||||
```
|
||||
|
||||
When trying to build test binary, one can often see linker complains about
|
||||
`undefined reference` to couple of symbols. This is one of solutions to
|
||||
determine all external dependencies of UUT - iteratively build test and resolve
|
||||
errors one by one. At this step, developer should decide either it's better to
|
||||
add an extra module to provide necessary definitions or rather mock such
|
||||
dependency. Quick guide through adding mocks is provided later in this doc.
|
||||
|
||||
## Writing new tests
|
||||
In coreboot, [Cmocka](https://cmocka.org/) is used as unit test framework. The
|
||||
project has exhaustive [API documentation](https://api.cmocka.org/). Let's see
|
||||
how we may incorporate it when writing tests.
|
||||
|
||||
### Assertions
|
||||
Testing the UUT consists of calling the functions in the UUT and comparing the
|
||||
returned values to the expected values. Cmocka implements
|
||||
[a set of assert macros](https://api.cmocka.org/group__cmocka__asserts.html) to
|
||||
compare a value with an expected value. If the two values do not match, the test
|
||||
fails with an error message.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
In our example, the simplest test is to call UUT for reading our fake devices
|
||||
registers and do all calculation in the test harness itself. At the end, let's
|
||||
compare integers with `assert_int_equal`.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
#define MASK 0x3
|
||||
#define SHIFT 0x1
|
||||
|
||||
static void i2c_read_field_test(void **state)
|
||||
{
|
||||
int bus, slave, reg;
|
||||
int i, j;
|
||||
uint8_t buf;
|
||||
|
||||
mock_expect_params_platform_i2c_transfer();
|
||||
|
||||
/* Read particular bits in all registers in all devices, then compare
|
||||
with expected value. */
|
||||
for (i = 0; i < ARRAY_SIZE(i2c_ex_devs); i++)
|
||||
for (j = 0; j < ARRAY_SIZE(i2c_ex_devs[0].regs); j++) {
|
||||
i2c_read_field(i2c_ex_devs[i].bus,
|
||||
i2c_ex_devs[i].slave,
|
||||
i2c_ex_devs[i].regs[j].reg,
|
||||
&buf, MASK, SHIFT);
|
||||
assert_int_equal((i2c_ex_devs[i].regs[j].data &
|
||||
(MASK << SHIFT)) >> SHIFT, buf);
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### Mocks
|
||||
|
||||
#### Overview
|
||||
Many coreboot modules are low level software that touch hardware directly.
|
||||
Because of this, one of the most important and challenging part of
|
||||
writing tests is to design and implement mocks. A mock is a software component
|
||||
which implements the API of another component so that the test can verify that
|
||||
certain functions are called (or not called), verify the parameters passed to
|
||||
those functions, and specify the return values from those functions. Mocks are
|
||||
especially useful when the API to be implemented is one that accesses hardware
|
||||
components.
|
||||
|
||||
When writing a mock, the developer implements the same API as the module being
|
||||
mocked. Such a mock may, for example, register a set of driver methods. Behind
|
||||
this API, there is usually a simulation of real hardware.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
For purpose of our i2c test, we may introduce two i2c devices with set of
|
||||
registers, which simply are structs in memory.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
/* Simulate two i2c devices, both on bus 0, each with three uint8_t regs
|
||||
implemented. */
|
||||
typedef struct {
|
||||
uint8_t reg;
|
||||
uint8_t data;
|
||||
} i2c_ex_regs_t;
|
||||
|
||||
typedef struct {
|
||||
unsigned int bus;
|
||||
uint8_t slave;
|
||||
i2c_ex_regs_t regs[3];
|
||||
} i2c_ex_devs_t;
|
||||
|
||||
i2c_ex_devs_t i2c_ex_devs[] = {
|
||||
{.bus = 0, .slave = 0xA, .regs = {
|
||||
{.reg = 0x0, .data = 0xB},
|
||||
{.reg = 0x1, .data = 0x6},
|
||||
{.reg = 0x2, .data = 0xF},
|
||||
} },
|
||||
{.bus = 0, .slave = 0x3, .regs = {
|
||||
{.reg = 0x0, .data = 0xDE},
|
||||
{.reg = 0x1, .data = 0xAD},
|
||||
{.reg = 0x2, .data = 0xBE},
|
||||
} },
|
||||
};
|
||||
|
||||
These fake devices will be accessed instead of hardware ones:
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
reg = tmp->buf[0];
|
||||
|
||||
/* Find object for requested device */
|
||||
for (i = 0; i < ARRAY_SIZE(i2c_ex_devs); i++, i2c_dev++)
|
||||
if (i2c_ex_devs[i].slave == tmp->slave) {
|
||||
i2c_dev = &i2c_ex_devs[i];
|
||||
break;
|
||||
}
|
||||
|
||||
if (i2c_dev == NULL)
|
||||
return -1;
|
||||
|
||||
/* Write commands */
|
||||
if (tmp->len > 1) {
|
||||
i2c_dev->regs[reg].data = tmp->buf[1];
|
||||
};
|
||||
|
||||
/* Read commands */
|
||||
for (i = 0; i < count; i++, tmp++)
|
||||
if (tmp->flags & I2C_M_RD) {
|
||||
*(tmp->buf) = i2c_dev->regs[reg].data;
|
||||
};
|
||||
```
|
||||
|
||||
Cmocka uses a feature that gcc provides for breaking dependencies at the link
|
||||
time. It is possible to override implementation of some function, with the
|
||||
method from test harness. This allows test harness to take control of execution
|
||||
from binary (during the execution of test), and stimulate UUT as required
|
||||
without changing the source code.
|
||||
|
||||
coreboot unit test infrastructure supports overriding of functions at link time.
|
||||
This is as simple as adding a `name_of_function` to be mocked into
|
||||
<test_name>-mocks variable in Makefile.inc. The result is that every time the
|
||||
function is called, `wrap_name_of_function` will be called instead.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
i2c-test-mocks += platform_i2c_transfer
|
||||
|
||||
Now, dev can write own implementation of `platform_i2c_transfer` and define it
|
||||
as `wrap_platform_i2c_transfer`. This implementation instead of accessing real
|
||||
i2c bus, will write/read from fake structs.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int __wrap_platform_i2c_transfer(unsigned int bus, struct i2c_msg *segments,
|
||||
int count)
|
||||
{
|
||||
}
|
||||
```
|
||||
|
||||
#### Checking mock's arguments
|
||||
A test can verify the parameters provided by the UUT to the mock function. The
|
||||
developer may also verify that number of calls to mock is correct and the order
|
||||
of calls to particular mocks is as expected (See
|
||||
[this](https://api.cmocka.org/group__cmocka__call__order.html)). The Cmocka
|
||||
macros for checking parameters are described
|
||||
[here](https://api.cmocka.org/group__cmocka__param.html). In general, in mock
|
||||
function, one makes a call to `check_expected(<param_name>)` and in the
|
||||
corresponding test function, `expect*()` macro, with description which parameter
|
||||
in which mock should have particular value, or be inside a described range.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
In our example, we may want to check that `platform_i2c_transfer` is fed with
|
||||
number of segments bigger than 0, each segment has flags which are in
|
||||
supported range and each segment has buf which is non-NULL. We are expecting
|
||||
such values for _every_ call, thus the last parameter in `expect*` macros is
|
||||
-1.
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
static void mock_expect_params_platform_i2c_transfer(void)
|
||||
{
|
||||
unsigned long int expected_flags[] = {0, I2C_M_RD, I2C_M_TEN,
|
||||
I2C_M_RECV_LEN, I2C_M_NOSTART};
|
||||
|
||||
/* Flags should always be only within supported range */
|
||||
expect_in_set_count(__wrap_platform_i2c_transfer, segments->flags,
|
||||
expected_flags, -1);
|
||||
|
||||
expect_not_value_count(__wrap_platform_i2c_transfer, segments->buf,
|
||||
NULL, -1);
|
||||
|
||||
expect_in_range_count(__wrap_platform_i2c_transfer, count, 1, INT_MAX,
|
||||
-1);
|
||||
}
|
||||
|
||||
And the checks below should be added to our mock
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
check_expected(count);
|
||||
|
||||
for (i = 0; i < count; i++, segments++) {
|
||||
check_expected_ptr(segments->buf);
|
||||
check_expected(segments->flags);
|
||||
}
|
||||
```
|
||||
|
||||
#### Instrument mocks
|
||||
It is possible for the test function to instrument what the mock will return to
|
||||
the UUT. This can be done by using the `will_return*()` and `mock()` macros.
|
||||
These are described in
|
||||
[the Mock Object section](https://api.cmocka.org/group__cmocka__mock.html) of
|
||||
the Cmocka API documentation.
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: Example
|
||||
|
||||
There is an non-coreboot example for using Cmocka available
|
||||
`here <https://lwn.net/Articles/558106/>`_.
|
||||
```
|
||||
|
||||
### Test runner
|
||||
Finally, the developer needs to implement the test `main()` function. All tests
|
||||
should be registered there and cmocka test runner invoked. All methods for
|
||||
invoking Cmocka test are described
|
||||
[here](https://api.cmocka.org/group__cmocka__exec.html).
|
||||
|
||||
```eval_rst
|
||||
.. admonition:: i2c-test example
|
||||
|
||||
We don't need any extra setup and teardown functions for i2c-test, so let's
|
||||
simply register test for `i2c_read_field` and return from main value which is
|
||||
output of Cmocka's runner (it returns number of tests that failed).
|
||||
|
||||
.. code-block:: c
|
||||
|
||||
int main(void)
|
||||
{
|
||||
const struct CMUnitTest tests[] = {
|
||||
cmocka_unit_test(i2c_read_field_test),
|
||||
};
|
||||
|
||||
return cmocka_run_group_tests(tests, NULL, NULL);
|
||||
}
|
||||
```
|
Loading…
Reference in New Issue