Provide add_timed_physim_test() in cmake

Add two new macros that allow to register physim tests for ctest,
including timing thresholds that should be fulfilled.

First, add_timed_physim_test() registers a new test using a helper
script RunTimedTest.cmake. This is because after test execution, we need
to analyze logs, and the recommended way to do multiple steps in one
test is via a helper cmake script:
https://cmake.org/pipermail/cmake-developers/2016-February/027816.html

Second, check_physim_threshold() adds new thresholds. It takes a text to
be parsed, and a condition (e.g., "< 20") to check for the number
following the threshold (which is assumed to be present right after the
threshold). It uses a test property to count the total number of checks
(limiting them to 10), and sets environment variables for the script.  I
initially planned to use a test property for checks, but those are only
valid in the same directory, and RunTimedTest.cmake seems to be assumed
by cmake to be "elsewhere", hence I needed to resort to environment
variables.

RunTimedTest.cmake is called through cmake with the test parameters and
checks. It re-constructs a list of checks [1], runs the test, and pipes
the log into a separate script that is passed all checks (see below).
Afterwards, it verifies that both the test and script passed.

A script analyze-timing.sh builds an awk script from the checks passed.
The script analyzes each line of the test output for the threshold, and
compares against the threshold. analyze-timing.sh returns success if all
checks passed.

[1] I did not manage to pass a "list" of checks in a single environment
variable through the check_physim_threshold(), which would be simpler.
This commit is contained in:
Robert Schmidt
2025-09-12 14:53:09 +02:00
parent 48be94a8c1
commit 55cebe6edf
4 changed files with 229 additions and 0 deletions
+38
View File
@@ -103,6 +103,7 @@ To define a new test or modify existing ones, update the following file:
openair1/SIMULATION/tests/CMakeLists.txt
```
## `add_physim_test()`
Use the `add_physim_test()` macro with the following arguments:
@@ -123,6 +124,43 @@ For instance, a PRACHsim looks like this:
These tests are run automatically as part of the following
pipelines: [RAN-PhySim-Cluster-4G](https://jenkins-oai.eurecom.fr/job/RAN-PhySim-Cluster-4G/) and [RAN-PhySim-Cluster-5G](https://jenkins-oai.eurecom.fr/job/RAN-PhySim-Cluster-5G/)
## `add_timed_physim_test()`
Use the `add_timed_physim_test()` macro to add a test the same way as with
`add_physim_test()` above. Additionally, it allows to check for thresholds with
`check_threshold()`:
check_threshold(<test_name> <threshold> <condition>)
where:
- `<test_name>` is any test that must have been added with
`add_timed_physim_test()`
- `<threshold>` is a threshold to check for, e.g., `PHY tx proc`, and
- `<condition>` a condition to check, e.g. `< 200`
There are two convenience functions to simplify the use of `check_threshold()`:
check_threshold_range(<test_name> <threshold> LOWER <lower> UPPER <upper>)
check_threshold_variance(<test_name> <threshold> AVG <avg> ABS_VAR <abs_var>)
where
- `<lower>` and `<upper>` are a lower and upper threshold, respectively, where
either one or both variables can be provided, and
- `<avg>` and `<abs_var>` are average and the variation in absolute numbers
(not a percentage!) can be provided.
Both functions internally use `check_threshold()`.
Thus upon execution of the test, the test will be run, but additionally ctest
will check for a match of `PHY tx proc <NUMBER>` (where `<NUMBER> is of format
`[0-9]+(\.[0-9]+)?`), and a matching number will be checked against condition
`< 200`.
For instance, this could look like this:
add_timed_physim_test(physim.5g.nr_dlsim.test3 "Some description" nr_dlsim -P)
check_physim_threshold(physim.5g.nr_dlsim.test3 "DLSCH encoding time" "< 50")
### How to rerun failed CI tests using `ctest`
Ctest automatically logs the failed tests in LastTestsFailed.log. This log is archived in