Skip to content

Testing

What every change proves before it merges. Host tests run on every pull request; the high and safety levels add sanitizers, coverage and cross-compilation.

Rule Level Applies to Verified
KB-TST-001 Host tests build and pass with gcc and clang must driver, library, hal, template checked
KB-TST-002 API misuse is proved to be a compile error must driver, library, template checked
KB-TST-003 Tests pass under AddressSanitizer and UndefinedBehaviorSanitizer must driver, library, hal, template (high level and above) checked
KB-TST-004 Coverage floor per assurance level must driver, library, hal (high level and above) checked
KB-TST-005 Compiles for every supported target must driver, library (high level and above) checked
KB-TST-006 Host tests compile every header with the firmware's flags and no platform paths must driver, library, template checked
KB-TST-007 Driver tests run against a register-level model of the chip must driver, template review
KB-TST-008 Host tests cover the failure paths, not only the happy path must driver, library, hal, template planned
KB-TST-009 Every driver ships a host example that CI builds and runs must driver, template checked
KB-TST-010 Every code block on the documentation site is compiled must driver, library, template planned
KB-TST-011 kb-core host tests run on every RTOS backend's host build must library planned
KB-TST-012 Every handler has a host test of its contract must library planned
KB-TST-013 A board HAL builds and is tested on the host must hal review
KB-TST-014 Firmware logic that needs no hardware is host-tested should firmware review
KB-TST-015 Tests are deterministic and independent of order must driver, library, hal, template planned
KB-TST-016 Every test has a timeout must driver, library, hal, template planned
KB-TST-017 Test files and cases name what they test should driver, library, hal, template planned
KB-TST-018 Every public function is exercised by a test should driver, library, hal, template planned
KB-TST-019 Share test setup through fixtures and helpers guide driver, library, hal, template none
KB-TST-020 Frame, CRC and protocol parsers are fuzzed must driver, library (high level and above) planned
KB-TST-021 Protocol and CRC code passes reference vectors should driver, library review
KB-TST-023 Hardware test runs leave a structured record must driver, hal, firmware (high level and above) planned
KB-TST-024 Real-time paths have a measured worst-case time and a regression check should driver, library, hal, firmware (high level and above) planned
KB-TST-025 Concurrent code is tested under ThreadSanitizer must library, hal (high level and above) planned
KB-TST-026 OTA updates are tested on both the update and the rollback path should firmware review
KB-TST-027 At safety level, every decision has full MC/DC coverage must driver, library, hal (safety level and above) checked
KB-TST-028 At safety level, every uncovered item is justified must driver, library, hal (safety level and above) planned
KB-TST-029 At safety level, mutation testing checks that the tests check something guide driver, library, hal (safety level and above) none
KB-TST-030 Hardware findings become model faults and tests; the model stands alone must driver, template planned

KB-TST-001: Host tests build and pass with gcc and clang

MUST · driver, library, hal, template · since 1.0

Every driver and library has unit tests that run on the host without hardware, against a mock port, and pass with both gcc and clang on every pull request.

Verified by:

  • host-tests (implemented): the unit tests build and pass on the host with gcc and clang

See also: IEC 62304:2006+A1:2015 5.5.5 (software unit verification)

KB-TST-002: API misuse is proved to be a compile error

MUST · driver, library, template · since 1.0

Each constraint the public API enforces at compile time (a port missing a hook, a wrong hook signature) has a test that must fail to compile, and CI checks that it does.

Verified by:

  • compile-fail-tests (implemented): a misuse of the public API is a compile error, proved by a test that must fail to compile

KB-TST-003: Tests pass under AddressSanitizer and UndefinedBehaviorSanitizer

MUST · driver, library, hal, template (high level and above) · since 1.0

The host tests also run in a build with -fsanitize=address,undefined and pass with no report.

Verified by:

  • sanitizers (implemented): host tests pass under AddressSanitizer and UndefinedBehaviorSanitizer

See also: SEI CERT C++ EXP53-CPP; SEI CERT C INT32-C

KB-TST-004: Coverage floor per assurance level

MUST · driver, library, hal (high level and above) · since 1.0

Coverage of library code by the host tests, measured with gcovr over a GCC -O0 --coverage build, stays at or above the floor of the repository's level: high, 85% of lines and 75% of branches; safety, 100% of reachable lines and branches. Tests, examples and compile-only targets are not counted.

Why: A floor that is defined by tool, build and scope can be reproduced by anyone; ADR-0020 says why these numbers.

Verified by:

  • coverage-floor/level (implemented): gcovr coverage of library code by the host tests at or above the floor (coverage-floor/level for lines and branches, coverage-floor/mcdc for condition coverage), as ADR-0020 defines it

See also: ADR-0020; IEC 62304:2006+A1:2015 5.5.5; IEC 61508-3:2010 Table B.2

KB-TST-005: Compiles for every supported target

MUST · driver, library (high level and above) · since 1.0

Drivers and libraries compile warning-free with arm-none-eabi-gcc (Cortex-M) and for ESP-IDF in CI, not only on the host.

Verified by:

  • cross-build/arm-none-eabi (implemented): the code compiles warning-free for the named target (arm-none-eabi Cortex-M cores, esp-idf, riscv32)
  • cross-build/esp-idf (implemented): the code compiles warning-free for the named target (arm-none-eabi Cortex-M cores, esp-idf, riscv32)

KB-TST-006: Host tests compile every header with the firmware's flags and no platform paths

MUST · driver, library, template · since 1.0

The host test build compiles every public header with -fno-exceptions -fno-rtti -Wall -Wextra -Werror (the template adds -Wpedantic -Wshadow -Wconversion -Wsign-conversion and an -O2 build) and with no ESP-IDF, vendor or board include path on the search path.

Why: A header that builds on the host with no platform in reach is proved MCU-agnostic.

Verified by:

  • host-tests (implemented): the unit tests build and pass on the host with gcc and clang
  • compiler-warnings/-Wall (implemented): the build is warning-free with the named flag and -Werror
  • compiler-warnings/-Wextra (implemented): the build is warning-free with the named flag and -Werror
  • compiler-warnings/-Wconversion (implemented): the build is warning-free with the named flag and -Werror
  • review: The test targets add no platform or vendor include directory.

See also: IEC 62304:2006+A1:2015 5.5.5

KB-TST-007: Driver tests run against a register-level model of the chip

MUST · driver, template · since 1.0

tests/mock_<chip>.hpp implements the driver's port as a model of the chip's registers and wire protocol, and every host test drives the driver through it. Drivers are not tested through virtual mocks of their own API.

Why: A register model catches wrong addresses, orders and bit masks; a mock of the API catches none of them.

Verified by:

  • review: tests/mock_<chip>.hpp models registers behind the CRTP port and all driver tests use it.

See also: ADR-0003

Explained in: standards/driver.md

KB-TST-008: Host tests cover the failure paths, not only the happy path

MUST · driver, library, hal, template · since 1.0

Host tests show that a call before EnsureInitialized() returns NotInitialized and sends nothing, a second EnsureInitialized() does no I/O, a failed start is retried, a wrong chip ID is refused, a failing bus hook is reported on every path that uses it, invalid arguments are rejected, and the expected register traffic is sent. New behaviour arrives with tests of its failure paths.

Verified by:

  • review: Every new error path, including a failing hook and a call before EnsureInitialized, has a test.
  • error-paths-tested (planned): every error value the public API can return is asserted by at least one host test

See also: IEC 62304:2006+A1:2015 5.5.4 (additional unit acceptance criteria: fault handling)

Explained in: standards/driver.md

KB-TST-009: Every driver ships a host example that CI builds and runs

MUST · driver, template · since 1.0

examples/host/ runs the driver on the host against the register model. It is built with the test flags and run as one of the host tests.

Verified by:

  • host-tests (implemented): the unit tests build and pass on the host with gcc and clang

See also: ADR-0011

Explained in: standards/driver.md

KB-TST-010: Every code block on the documentation site is compiled

MUST · driver, library, template · since 1.0

C++ shown on the documentation site is included from tests/docs/, and the host test build compiles those files against the current headers.

Verified by:

  • doc-snippets (planned): every C++ block on the documentation site is a snippet from tests/docs/ and those files compile against the current headers (driver-template doc_snippets ctest and the kokeb-docs check)
  • review: New code blocks on the site are snippets from tests/docs/, not inline code.

Explained in: standards/docs.md

KB-TST-011: kb-core host tests run on every RTOS backend's host build

Proposed

Warns until 2027-03-01, then fails CI.

MUST · library · since 1.0 · enforced from 2027-03-01

kb-core's host tests live in tests/host/ and run with gcc and clang for KB_RTOS=FREERTOS (against the host shim) and KB_RTOS=NONE, with KB_MCU=HOST.

Verified by:

  • host-tests-matrix/kb-rtos (planned): the host tests pass with gcc and clang for each configuration of the named matrix (kb-rtos: KB_RTOS=FREERTOS host shim and KB_RTOS=NONE, with KB_MCU=HOST)

See also: ADR-0013; IEC 62304:2006+A1:2015 5.5.5

Explained in: standards/library.md

KB-TST-012: Every handler has a host test of its contract

MUST · library · since 1.0

Each handler has tests/host/test_<chip>_handler.cpp, run with KB_RTOS=NONE and KB_MCU=HOST, showing that construction does no I/O, retries happen as documented, a bus failure maps to Status::Bus, VisitDriver refuses when it should, and the diagnostics counters count.

Verified by:

  • test-presence/handler (planned): every unit of the named kind has a test (handler: tests/host/test_<chip>_handler.cpp; public-function: each public function is called by a host test)
  • review: The handler test covers construction, retry, bus errors, VisitDriver and diagnostics.

Explained in: standards/handler.md

KB-TST-013: A board HAL builds and is tested on the host

MUST · hal · since 1.0

A HAL builds on the host with its host bind, and its tests cover the start-up order, that each manager fails closed, and CollectDiagnostics. pincfg builds for the host with an empty pin table.

Verified by:

  • review: The HAL host tests cover start-up order, fail-closed managers and diagnostics.

Explained in: standards/hal.md

KB-TST-014: Firmware logic that needs no hardware is host-tested

SHOULD · firmware · since 1.0

Application logic that does not touch hardware has host tests that run against the HAL's host bind.

Verified by:

  • review: New hardware-free application logic has host tests.

Explained in: standards/product.md

KB-TST-015: Tests are deterministic and independent of order

Proposed

Warns until 2027-03-01, then fails CI.

MUST · driver, library, hal, template · since 1.0 · enforced from 2027-03-01

A test gives the same result on every run and in any order: no unseeded randomness, wall-clock dependence or state shared between tests. CI runs the suite repeatedly and in random order.

Why: A flaky test teaches people to re-run CI instead of reading it.

Verified by:

  • test-repeat (planned): ctest --repeat until-fail:20 --schedule-random passes, so no test depends on luck or order

KB-TST-016: Every test has a timeout

Proposed

Warns until 2027-03-01, then fails CI.

MUST · driver, library, hal, template · since 1.0 · enforced from 2027-03-01

Every ctest test has a TIMEOUT property, so a hung test fails in bounded time instead of stalling CI.

Verified by:

  • ctest-timeouts (planned): every test ctest lists has a TIMEOUT property

KB-TST-017: Test files and cases name what they test

Proposed

Warns until 2027-03-01, then fails CI.

SHOULD · driver, library, hal, template · since 1.0 · enforced from 2027-03-01

Host test files are named test_<unit>.cpp after the unit they exercise, and each test case name states the behaviour it checks.

Verified by:

  • test-naming (planned): host test sources are named test_<unit>.cpp after an existing unit of the library

KB-TST-018: Every public function is exercised by a test

Proposed

Warns until 2027-03-01, then fails CI.

SHOULD · driver, library, hal, template · since 1.0 · enforced from 2027-03-01

Each function of the public API is called by at least one host test.

Verified by:

  • test-presence/public-function (planned): every unit of the named kind has a test (handler: tests/host/test_<chip>_handler.cpp; public-function: each public function is called by a host test)

See also: IEC 62304:2006+A1:2015 5.5.5

KB-TST-019: Share test setup through fixtures and helpers

GUIDE · driver, library, hal, template · since 1.0

Setup used by more than one test lives in a fixture or a helper in tests/, not copied into each test.

KB-TST-020: Frame, CRC and protocol parsers are fuzzed

Proposed

Warns until 2027-03-01, then fails CI.

MUST · driver, library (high level and above) · since 1.0 · enforced from 2027-03-01

Every function that parses bytes from a bus or a host (frames, CRCs, protocol messages) has a libFuzzer target, built with the sanitizers, that CI runs for a fixed time on each pull request; its corpus is committed and every crash it found is kept as a regression test.

Why: Parsers face input nobody wrote a test for; a fuzzer writes those tests.

Verified by:

  • fuzzing (planned): each parser's libFuzzer target runs on the host for a fixed time under ASan and UBSan without a finding

See also: SEI CERT C INT32-C; SEI CERT C++ CTR50-CPP

KB-TST-021: Protocol and CRC code passes reference vectors

SHOULD · driver, library · since 1.0

An implementation of a published protocol or CRC is tested against the reference vectors of its specification or datasheet, with the source of each vector named in the test.

Verified by:

  • review: Protocol and CRC tests use vectors from the specification and cite it.

KB-TST-022: Every MCU port passes the interface conformance tests

Retired

Duplicate of KB-HW-002, merged when the rules were loaded. Replaced by KB-HW-002.

MUST · library, hal · since 1.0

kb::interfaces has a conformance suite per interface, and every mcu/<family>/ port runs it. A new interface method arrives with its conformance test.

Verified by:

  • conformance-tests (planned): every kb::interfaces method has a conformance test, and each mcu/<family> port runs and passes the suite
  • review: A new interface method adds a conformance test.

See also: ADR-0014; IEC 62304:2006+A1:2015 5.6 (integration testing)

KB-TST-023: Hardware test runs leave a structured record

Proposed

Warns until 2027-03-01, then fails CI.

MUST · driver, hal, firmware (high level and above) · since 1.0 · enforced from 2027-03-01

Tests that need a board live in tests/hardware/. Each run on a bench or a self-hosted runner writes a record of the commit SHA, board and revision, toolchain, test list, results and date, kept with the release; a change not run on hardware says so in its pull request.

Verified by:

  • hil-evidence (planned): a hardware test record (commit, board and revision, toolchain, tests, results, date) exists for the release and each supported board, in the evidence format

See also: IEC 62304:2006+A1:2015 5.7 (software system testing records)

Explained in: evidence.md

KB-TST-024: Real-time paths have a measured worst-case time and a regression check

Proposed

Warns until 2027-03-01, then fails CI.

SHOULD · driver, library, hal, firmware (high level and above) · since 1.0 · enforced from 2027-03-01

A function with a timing requirement documents its worst-case execution time measured on the target, and a benchmark job compares it with the stored baseline and fails on a regression beyond the recorded margin.

Verified by:

  • timing-budget (planned): an on-target benchmark job compares measured worst-case times with the stored baseline and fails beyond its margin

KB-TST-025: Concurrent code is tested under ThreadSanitizer

Proposed

Warns until 2027-03-01, then fails CI.

MUST · library, hal (high level and above) · since 1.0 · enforced from 2027-03-01

Code that shares state between tasks (kb::rtos, thread-safe managers) has host stress tests with many concurrent callers that pass under -fsanitize=thread, with no failure attributed to contention.

Verified by:

  • tsan (planned): the concurrency host tests pass under ThreadSanitizer

KB-TST-026: OTA updates are tested on both the update and the rollback path

SHOULD · firmware · since 1.0

Before firmware with over-the-air update ships, a good update and a failed update that rolls back are both run on hardware and recorded in the release checklist.

Verified by:

  • review: The release checklist records an OTA update and an OTA rollback on hardware.

KB-TST-027: At safety level, every decision has full MC/DC coverage

MUST · driver, library, hal (safety level and above) · since 1.0

Library code reaches 100% modified condition/decision coverage, measured with GCC's -fcondition-coverage and gcovr; at the high level the figure is reported but not gated.

Why: Branch coverage can pass while a condition inside a compound decision never decided anything; MC/DC shows each condition independently changes the outcome.

Verified by:

  • coverage-floor/mcdc (implemented): gcovr coverage of library code by the host tests at or above the floor (coverage-floor/level for lines and branches, coverage-floor/mcdc for condition coverage), as ADR-0020 defines it

See also: ADR-0020; IEC 62304:2006+A1:2015 5.5.4 (additional acceptance criteria for class C)

KB-TST-028: At safety level, every uncovered item is justified

Proposed

Warns until 2027-03-01, then fails CI.

MUST · driver, library, hal (safety level and above) · since 1.0 · enforced from 2027-03-01

Each line, branch or condition the tests do not reach is listed in docs/coverage-justification.yml with the reason and who accepted it; an entry for an item the tests now reach is removed.

Verified by:

  • coverage-justification (planned): every uncovered line, branch or condition is in docs/coverage-justification.yml, and nothing else is

See also: ADR-0020

Explained in: evidence.md

KB-TST-029: At safety level, mutation testing checks that the tests check something

GUIDE · driver, library, hal (safety level and above) · since 1.0

Each release of a safety component runs mutation testing (for example Mull) over library code and reviews the surviving mutants; each survivor gets a test or a note on why it is equivalent.

Why: Coverage shows which code ran, not whether a test would notice it being wrong.

See also: ADR-0020

KB-TST-030: Hardware findings become model faults and tests; the model stands alone

Proposed

Warns until 2027-03-01, then fails CI. Extends KB-TST-007.

MUST · driver, template · since 1.0 · enforced from 2027-03-01

Each behaviour found on hardware becomes a fault mode of tests/mock_<chip>.hpp and a regression test that fails on the code before the fix; the test cites the datasheet, and private evidence stays outside the repository. The model is written from the datasheet on its own: it does not include the driver's register header but carries its own register map with datasheet page references, and a test cross-checks the two maps with static_assert. A chip with timing (busy windows, settling, wake-up, pipeline latency) has a virtual clock in its model.

Why: A model that includes the driver's register header shares its mistakes: a wrong constant hides behind the same wrong constant. Without a model fault, a fix found on the bench cannot be proven or kept.

Verified by:

  • model-virtual-clock (planned): a chip model for a driver whose port has DelayUs or DelayMs advances a virtual clock in those hooks, and a test runs a delayed-ready mode against it
  • review: The pull request answers "hardware finding: model fault and test?"; tests/mock_<chip>.hpp includes no driver register header, and a test static_asserts the model's map against the driver's.

Explained in: standards/driver.md