Testing¶
What every change proves before it merges. Host tests run on every pull request; the high and safety levels add sanitizers, coverage and cross-compilation.
| Rule | Level | Applies to | Verified |
|---|---|---|---|
| KB-TST-001 Host tests build and pass with gcc and clang | must | driver, library, hal, template | checked |
| KB-TST-002 API misuse is proved to be a compile error | must | driver, library, template | checked |
| KB-TST-003 Tests pass under AddressSanitizer and UndefinedBehaviorSanitizer | must | driver, library, hal, template (high level and above) | checked |
| KB-TST-004 Coverage floor per assurance level | must | driver, library, hal (high level and above) | checked |
| KB-TST-005 Compiles for every supported target | must | driver, library (high level and above) | checked |
| KB-TST-006 Host tests compile every header with the firmware's flags and no platform paths | must | driver, library, template | checked |
| KB-TST-007 Driver tests run against a register-level model of the chip | must | driver, template | review |
| KB-TST-008 Host tests cover the failure paths, not only the happy path | must | driver, library, hal, template | planned |
| KB-TST-009 Every driver ships a host example that CI builds and runs | must | driver, template | checked |
| KB-TST-010 Every code block on the documentation site is compiled | must | driver, library, template | planned |
| KB-TST-011 kb-core host tests run on every RTOS backend's host build | must | library | planned |
| KB-TST-012 Every handler has a host test of its contract | must | library | planned |
| KB-TST-013 A board HAL builds and is tested on the host | must | hal | review |
| KB-TST-014 Firmware logic that needs no hardware is host-tested | should | firmware | review |
| KB-TST-015 Tests are deterministic and independent of order | must | driver, library, hal, template | planned |
| KB-TST-016 Every test has a timeout | must | driver, library, hal, template | planned |
| KB-TST-017 Test files and cases name what they test | should | driver, library, hal, template | planned |
| KB-TST-018 Every public function is exercised by a test | should | driver, library, hal, template | planned |
| KB-TST-019 Share test setup through fixtures and helpers | guide | driver, library, hal, template | none |
| KB-TST-020 Frame, CRC and protocol parsers are fuzzed | must | driver, library (high level and above) | planned |
| KB-TST-021 Protocol and CRC code passes reference vectors | should | driver, library | review |
| KB-TST-023 Hardware test runs leave a structured record | must | driver, hal, firmware (high level and above) | planned |
| KB-TST-024 Real-time paths have a measured worst-case time and a regression check | should | driver, library, hal, firmware (high level and above) | planned |
| KB-TST-025 Concurrent code is tested under ThreadSanitizer | must | library, hal (high level and above) | planned |
| KB-TST-026 OTA updates are tested on both the update and the rollback path | should | firmware | review |
| KB-TST-027 At safety level, every decision has full MC/DC coverage | must | driver, library, hal (safety level and above) | checked |
| KB-TST-028 At safety level, every uncovered item is justified | must | driver, library, hal (safety level and above) | planned |
| KB-TST-029 At safety level, mutation testing checks that the tests check something | guide | driver, library, hal (safety level and above) | none |
| KB-TST-030 Hardware findings become model faults and tests; the model stands alone | must | driver, template | planned |
KB-TST-001: Host tests build and pass with gcc and clang¶
MUST · driver, library, hal, template · since 1.0
Every driver and library has unit tests that run on the host without hardware, against a mock port, and pass with both gcc and clang on every pull request.
Verified by:
host-tests(implemented): the unit tests build and pass on the host with gcc and clang
See also: IEC 62304:2006+A1:2015 5.5.5 (software unit verification)
KB-TST-002: API misuse is proved to be a compile error¶
MUST · driver, library, template · since 1.0
Each constraint the public API enforces at compile time (a port missing a hook, a wrong hook signature) has a test that must fail to compile, and CI checks that it does.
Verified by:
compile-fail-tests(implemented): a misuse of the public API is a compile error, proved by a test that must fail to compile
KB-TST-003: Tests pass under AddressSanitizer and UndefinedBehaviorSanitizer¶
MUST · driver, library, hal, template (high level and above) · since 1.0
The host tests also run in a build with -fsanitize=address,undefined and pass with no report.
Verified by:
sanitizers(implemented): host tests pass under AddressSanitizer and UndefinedBehaviorSanitizer
See also: SEI CERT C++ EXP53-CPP; SEI CERT C INT32-C
KB-TST-004: Coverage floor per assurance level¶
MUST · driver, library, hal (high level and above) · since 1.0
Coverage of library code by the host tests, measured with gcovr over a GCC -O0 --coverage build, stays at or above the
floor of the repository's level: high, 85% of lines and 75% of branches; safety, 100% of reachable lines and branches.
Tests, examples and compile-only targets are not counted.
Why: A floor that is defined by tool, build and scope can be reproduced by anyone; ADR-0020 says why these numbers.
Verified by:
coverage-floor/level(implemented): gcovr coverage of library code by the host tests at or above the floor (coverage-floor/level for lines and branches, coverage-floor/mcdc for condition coverage), as ADR-0020 defines it
See also: ADR-0020; IEC 62304:2006+A1:2015 5.5.5; IEC 61508-3:2010 Table B.2
KB-TST-005: Compiles for every supported target¶
MUST · driver, library (high level and above) · since 1.0
Drivers and libraries compile warning-free with arm-none-eabi-gcc (Cortex-M) and for ESP-IDF in CI, not only on the host.
Verified by:
cross-build/arm-none-eabi(implemented): the code compiles warning-free for the named target (arm-none-eabi Cortex-M cores, esp-idf, riscv32)cross-build/esp-idf(implemented): the code compiles warning-free for the named target (arm-none-eabi Cortex-M cores, esp-idf, riscv32)
KB-TST-006: Host tests compile every header with the firmware's flags and no platform paths¶
MUST · driver, library, template · since 1.0
The host test build compiles every public header with -fno-exceptions -fno-rtti -Wall -Wextra -Werror (the template
adds -Wpedantic -Wshadow -Wconversion -Wsign-conversion and an -O2 build) and with no ESP-IDF, vendor or board
include path on the search path.
Why: A header that builds on the host with no platform in reach is proved MCU-agnostic.
Verified by:
host-tests(implemented): the unit tests build and pass on the host with gcc and clangcompiler-warnings/-Wall(implemented): the build is warning-free with the named flag and -Werrorcompiler-warnings/-Wextra(implemented): the build is warning-free with the named flag and -Werrorcompiler-warnings/-Wconversion(implemented): the build is warning-free with the named flag and -Werror- review: The test targets add no platform or vendor include directory.
See also: IEC 62304:2006+A1:2015 5.5.5
KB-TST-007: Driver tests run against a register-level model of the chip¶
MUST · driver, template · since 1.0
tests/mock_<chip>.hpp implements the driver's port as a model of the chip's registers and wire protocol, and every
host test drives the driver through it. Drivers are not tested through virtual mocks of their own API.
Why: A register model catches wrong addresses, orders and bit masks; a mock of the API catches none of them.
Verified by:
- review: tests/mock_<chip>.hpp models registers behind the CRTP port and all driver tests use it.
See also: ADR-0003
Explained in: standards/driver.md
KB-TST-008: Host tests cover the failure paths, not only the happy path¶
MUST · driver, library, hal, template · since 1.0
Host tests show that a call before EnsureInitialized() returns NotInitialized and sends nothing, a second
EnsureInitialized() does no I/O, a failed start is retried, a wrong chip ID is refused, a failing bus hook is reported
on every path that uses it, invalid arguments are rejected, and the expected register traffic is sent. New behaviour
arrives with tests of its failure paths.
Verified by:
- review: Every new error path, including a failing hook and a call before EnsureInitialized, has a test.
error-paths-tested(planned): every error value the public API can return is asserted by at least one host test
See also: IEC 62304:2006+A1:2015 5.5.4 (additional unit acceptance criteria: fault handling)
Explained in: standards/driver.md
KB-TST-009: Every driver ships a host example that CI builds and runs¶
MUST · driver, template · since 1.0
examples/host/ runs the driver on the host against the register model. It is built with the test flags and run as one
of the host tests.
Verified by:
host-tests(implemented): the unit tests build and pass on the host with gcc and clang
See also: ADR-0011
Explained in: standards/driver.md
KB-TST-010: Every code block on the documentation site is compiled¶
MUST · driver, library, template · since 1.0
C++ shown on the documentation site is included from tests/docs/, and the host test build compiles those files against
the current headers.
Verified by:
doc-snippets(planned): every C++ block on the documentation site is a snippet from tests/docs/ and those files compile against the current headers (driver-template doc_snippets ctest and the kokeb-docs check)- review: New code blocks on the site are snippets from tests/docs/, not inline code.
Explained in: standards/docs.md
KB-TST-011: kb-core host tests run on every RTOS backend's host build¶
Proposed
Warns until 2027-03-01, then fails CI.
MUST · library · since 1.0 · enforced from 2027-03-01
kb-core's host tests live in tests/host/ and run with gcc and clang for KB_RTOS=FREERTOS (against the host shim) and
KB_RTOS=NONE, with KB_MCU=HOST.
Verified by:
host-tests-matrix/kb-rtos(planned): the host tests pass with gcc and clang for each configuration of the named matrix (kb-rtos: KB_RTOS=FREERTOS host shim and KB_RTOS=NONE, with KB_MCU=HOST)
See also: ADR-0013; IEC 62304:2006+A1:2015 5.5.5
Explained in: standards/library.md
KB-TST-012: Every handler has a host test of its contract¶
MUST · library · since 1.0
Each handler has tests/host/test_<chip>_handler.cpp, run with KB_RTOS=NONE and KB_MCU=HOST, showing that
construction does no I/O, retries happen as documented, a bus failure maps to Status::Bus, VisitDriver refuses when
it should, and the diagnostics counters count.
Verified by:
test-presence/handler(planned): every unit of the named kind has a test (handler: tests/host/test_<chip>_handler.cpp; public-function: each public function is called by a host test)- review: The handler test covers construction, retry, bus errors, VisitDriver and diagnostics.
Explained in: standards/handler.md
KB-TST-013: A board HAL builds and is tested on the host¶
MUST · hal · since 1.0
A HAL builds on the host with its host bind, and its tests cover the start-up order, that each manager fails closed, and
CollectDiagnostics. pincfg builds for the host with an empty pin table.
Verified by:
- review: The HAL host tests cover start-up order, fail-closed managers and diagnostics.
Explained in: standards/hal.md
KB-TST-014: Firmware logic that needs no hardware is host-tested¶
SHOULD · firmware · since 1.0
Application logic that does not touch hardware has host tests that run against the HAL's host bind.
Verified by:
- review: New hardware-free application logic has host tests.
Explained in: standards/product.md
KB-TST-015: Tests are deterministic and independent of order¶
Proposed
Warns until 2027-03-01, then fails CI.
MUST · driver, library, hal, template · since 1.0 · enforced from 2027-03-01
A test gives the same result on every run and in any order: no unseeded randomness, wall-clock dependence or state shared between tests. CI runs the suite repeatedly and in random order.
Why: A flaky test teaches people to re-run CI instead of reading it.
Verified by:
test-repeat(planned): ctest --repeat until-fail:20 --schedule-random passes, so no test depends on luck or order
KB-TST-016: Every test has a timeout¶
Proposed
Warns until 2027-03-01, then fails CI.
MUST · driver, library, hal, template · since 1.0 · enforced from 2027-03-01
Every ctest test has a TIMEOUT property, so a hung test fails in bounded time instead of stalling CI.
Verified by:
ctest-timeouts(planned): every test ctest lists has a TIMEOUT property
KB-TST-017: Test files and cases name what they test¶
Proposed
Warns until 2027-03-01, then fails CI.
SHOULD · driver, library, hal, template · since 1.0 · enforced from 2027-03-01
Host test files are named test_<unit>.cpp after the unit they exercise, and each test case name states the behaviour
it checks.
Verified by:
test-naming(planned): host test sources are named test_<unit>.cpp after an existing unit of the library
KB-TST-018: Every public function is exercised by a test¶
Proposed
Warns until 2027-03-01, then fails CI.
SHOULD · driver, library, hal, template · since 1.0 · enforced from 2027-03-01
Each function of the public API is called by at least one host test.
Verified by:
test-presence/public-function(planned): every unit of the named kind has a test (handler: tests/host/test_<chip>_handler.cpp; public-function: each public function is called by a host test)
See also: IEC 62304:2006+A1:2015 5.5.5
KB-TST-019: Share test setup through fixtures and helpers¶
GUIDE · driver, library, hal, template · since 1.0
Setup used by more than one test lives in a fixture or a helper in tests/, not copied into each test.
KB-TST-020: Frame, CRC and protocol parsers are fuzzed¶
Proposed
Warns until 2027-03-01, then fails CI.
MUST · driver, library (high level and above) · since 1.0 · enforced from 2027-03-01
Every function that parses bytes from a bus or a host (frames, CRCs, protocol messages) has a libFuzzer target, built with the sanitizers, that CI runs for a fixed time on each pull request; its corpus is committed and every crash it found is kept as a regression test.
Why: Parsers face input nobody wrote a test for; a fuzzer writes those tests.
Verified by:
fuzzing(planned): each parser's libFuzzer target runs on the host for a fixed time under ASan and UBSan without a finding
See also: SEI CERT C INT32-C; SEI CERT C++ CTR50-CPP
KB-TST-021: Protocol and CRC code passes reference vectors¶
SHOULD · driver, library · since 1.0
An implementation of a published protocol or CRC is tested against the reference vectors of its specification or datasheet, with the source of each vector named in the test.
Verified by:
- review: Protocol and CRC tests use vectors from the specification and cite it.
KB-TST-022: Every MCU port passes the interface conformance tests¶
Retired
Duplicate of KB-HW-002, merged when the rules were loaded. Replaced by KB-HW-002.
MUST · library, hal · since 1.0
kb::interfaces has a conformance suite per interface, and every mcu/<family>/ port runs it. A new interface method
arrives with its conformance test.
Verified by:
conformance-tests(planned): every kb::interfaces method has a conformance test, and each mcu/<family> port runs and passes the suite- review: A new interface method adds a conformance test.
See also: ADR-0014; IEC 62304:2006+A1:2015 5.6 (integration testing)
KB-TST-023: Hardware test runs leave a structured record¶
Proposed
Warns until 2027-03-01, then fails CI.
MUST · driver, hal, firmware (high level and above) · since 1.0 · enforced from 2027-03-01
Tests that need a board live in tests/hardware/. Each run on a bench or a self-hosted runner writes a record of the
commit SHA, board and revision, toolchain, test list, results and date, kept with the release; a change not run on
hardware says so in its pull request.
Verified by:
hil-evidence(planned): a hardware test record (commit, board and revision, toolchain, tests, results, date) exists for the release and each supported board, in the evidence format
See also: IEC 62304:2006+A1:2015 5.7 (software system testing records)
Explained in: evidence.md
KB-TST-024: Real-time paths have a measured worst-case time and a regression check¶
Proposed
Warns until 2027-03-01, then fails CI.
SHOULD · driver, library, hal, firmware (high level and above) · since 1.0 · enforced from 2027-03-01
A function with a timing requirement documents its worst-case execution time measured on the target, and a benchmark job compares it with the stored baseline and fails on a regression beyond the recorded margin.
Verified by:
timing-budget(planned): an on-target benchmark job compares measured worst-case times with the stored baseline and fails beyond its margin
KB-TST-025: Concurrent code is tested under ThreadSanitizer¶
Proposed
Warns until 2027-03-01, then fails CI.
MUST · library, hal (high level and above) · since 1.0 · enforced from 2027-03-01
Code that shares state between tasks (kb::rtos, thread-safe managers) has host stress tests with many concurrent callers
that pass under -fsanitize=thread, with no failure attributed to contention.
Verified by:
tsan(planned): the concurrency host tests pass under ThreadSanitizer
KB-TST-026: OTA updates are tested on both the update and the rollback path¶
SHOULD · firmware · since 1.0
Before firmware with over-the-air update ships, a good update and a failed update that rolls back are both run on hardware and recorded in the release checklist.
Verified by:
- review: The release checklist records an OTA update and an OTA rollback on hardware.
KB-TST-027: At safety level, every decision has full MC/DC coverage¶
MUST · driver, library, hal (safety level and above) · since 1.0
Library code reaches 100% modified condition/decision coverage, measured with GCC's -fcondition-coverage and gcovr; at
the high level the figure is reported but not gated.
Why: Branch coverage can pass while a condition inside a compound decision never decided anything; MC/DC shows each condition independently changes the outcome.
Verified by:
coverage-floor/mcdc(implemented): gcovr coverage of library code by the host tests at or above the floor (coverage-floor/level for lines and branches, coverage-floor/mcdc for condition coverage), as ADR-0020 defines it
See also: ADR-0020; IEC 62304:2006+A1:2015 5.5.4 (additional acceptance criteria for class C)
KB-TST-028: At safety level, every uncovered item is justified¶
Proposed
Warns until 2027-03-01, then fails CI.
MUST · driver, library, hal (safety level and above) · since 1.0 · enforced from 2027-03-01
Each line, branch or condition the tests do not reach is listed in docs/coverage-justification.yml with the reason and
who accepted it; an entry for an item the tests now reach is removed.
Verified by:
coverage-justification(planned): every uncovered line, branch or condition is in docs/coverage-justification.yml, and nothing else is
See also: ADR-0020
Explained in: evidence.md
KB-TST-029: At safety level, mutation testing checks that the tests check something¶
GUIDE · driver, library, hal (safety level and above) · since 1.0
Each release of a safety component runs mutation testing (for example Mull) over library code and reviews the surviving mutants; each survivor gets a test or a note on why it is equivalent.
Why: Coverage shows which code ran, not whether a test would notice it being wrong.
See also: ADR-0020
KB-TST-030: Hardware findings become model faults and tests; the model stands alone¶
Proposed
Warns until 2027-03-01, then fails CI. Extends KB-TST-007.
MUST · driver, template · since 1.0 · enforced from 2027-03-01
Each behaviour found on hardware becomes a fault mode of tests/mock_<chip>.hpp and a regression test that fails on the
code before the fix; the test cites the datasheet, and private evidence stays outside the repository. The model is
written from the datasheet on its own: it does not include the driver's register header but carries its own register map
with datasheet page references, and a test cross-checks the two maps with static_assert. A chip with timing (busy
windows, settling, wake-up, pipeline latency) has a virtual clock in its model.
Why: A model that includes the driver's register header shares its mistakes: a wrong constant hides behind the same wrong constant. Without a model fault, a fix found on the bench cannot be proven or kept.
Verified by:
model-virtual-clock(planned): a chip model for a driver whose port has DelayUs or DelayMs advances a virtual clock in those hooks, and a test runs a delayed-ready mode against it- review: The pull request answers "hardware finding: model fault and test?"; tests/mock_<chip>.hpp includes no driver register header, and a test static_asserts the model's map against the driver's.
Explained in: standards/driver.md