Provide add_timed_physim_test() in cmake
Add two new macros that allow to register physim tests for ctest, including timing thresholds that should be fulfilled. First, add_timed_physim_test() registers a new test using a helper script RunTimedTest.cmake. This is because after test execution, we need to analyze logs, and the recommended way to do multiple steps in one test is via a helper cmake script: https://cmake.org/pipermail/cmake-developers/2016-February/027816.html Second, check_physim_threshold() adds new thresholds. It takes a text to be parsed, and a condition (e.g., "< 20") to check for the number following the threshold (which is assumed to be present right after the threshold). It uses a test property to count the total number of checks (limiting them to 10), and sets environment variables for the script. I initially planned to use a test property for checks, but those are only valid in the same directory, and RunTimedTest.cmake seems to be assumed by cmake to be "elsewhere", hence I needed to resort to environment variables. RunTimedTest.cmake is called through cmake with the test parameters and checks. It re-constructs a list of checks [1], runs the test, and pipes the log into a separate script that is passed all checks (see below). Afterwards, it verifies that both the test and script passed. A script analyze-timing.sh builds an awk script from the checks passed. The script analyzes each line of the test output for the threshold, and compares against the threshold. analyze-timing.sh returns success if all checks passed. [1] I did not manage to pass a "list" of checks in a single environment variable through the check_physim_threshold(), which would be simpler.
This commit is contained in:
@@ -103,6 +103,7 @@ To define a new test or modify existing ones, update the following file:
|
||||
openair1/SIMULATION/tests/CMakeLists.txt
|
||||
```
|
||||
|
||||
## `add_physim_test()`
|
||||
|
||||
Use the `add_physim_test()` macro with the following arguments:
|
||||
|
||||
@@ -123,6 +124,43 @@ For instance, a PRACHsim looks like this:
|
||||
These tests are run automatically as part of the following
|
||||
pipelines: [RAN-PhySim-Cluster-4G](https://jenkins-oai.eurecom.fr/job/RAN-PhySim-Cluster-4G/) and [RAN-PhySim-Cluster-5G](https://jenkins-oai.eurecom.fr/job/RAN-PhySim-Cluster-5G/)
|
||||
|
||||
## `add_timed_physim_test()`
|
||||
|
||||
Use the `add_timed_physim_test()` macro to add a test the same way as with
|
||||
`add_physim_test()` above. Additionally, it allows to check for thresholds with
|
||||
`check_threshold()`:
|
||||
|
||||
check_threshold(<test_name> <threshold> <condition>)
|
||||
|
||||
where:
|
||||
- `<test_name>` is any test that must have been added with
|
||||
`add_timed_physim_test()`
|
||||
- `<threshold>` is a threshold to check for, e.g., `PHY tx proc`, and
|
||||
- `<condition>` a condition to check, e.g. `< 200`
|
||||
|
||||
There are two convenience functions to simplify the use of `check_threshold()`:
|
||||
|
||||
check_threshold_range(<test_name> <threshold> LOWER <lower> UPPER <upper>)
|
||||
check_threshold_variance(<test_name> <threshold> AVG <avg> ABS_VAR <abs_var>)
|
||||
|
||||
where
|
||||
- `<lower>` and `<upper>` are a lower and upper threshold, respectively, where
|
||||
either one or both variables can be provided, and
|
||||
- `<avg>` and `<abs_var>` are average and the variation in absolute numbers
|
||||
(not a percentage!) can be provided.
|
||||
|
||||
Both functions internally use `check_threshold()`.
|
||||
|
||||
Thus upon execution of the test, the test will be run, but additionally ctest
|
||||
will check for a match of `PHY tx proc <NUMBER>` (where `<NUMBER> is of format
|
||||
`[0-9]+(\.[0-9]+)?`), and a matching number will be checked against condition
|
||||
`< 200`.
|
||||
|
||||
For instance, this could look like this:
|
||||
|
||||
add_timed_physim_test(physim.5g.nr_dlsim.test3 "Some description" nr_dlsim -P)
|
||||
check_physim_threshold(physim.5g.nr_dlsim.test3 "DLSCH encoding time" "< 50")
|
||||
|
||||
### How to rerun failed CI tests using `ctest`
|
||||
|
||||
Ctest automatically logs the failed tests in LastTestsFailed.log. This log is archived in
|
||||
|
||||
Reference in New Issue
Block a user