Replace the previous add_timed_physim_test() macro with add_physim_test()
to use a single, unified function for defining physim tests. This simplifies
the test configuration and removes redundant macro definitions while keeping
all existing functionality.
This change follows up on the earlier addition of support for including
custom timing threshold files (PHYSIM_CHECK_FILES).
Update existing tests accordingly.
For some physical simulators, we have multiple SNR iterations.
The previous version of the script correctly retained only the last
measurement, but if a previous measurement condition set RC=1, the
script would fail although the final "good" run might fulfil the
measurement condition (IOW, it would show SUCCESS, but still fail on the
check).
Instead, check if all measurement lines finish on SUCCESS, which handles
the above case.
The awk script in analyze-timing.sh used syntax that works with gawk but fails
on systems where mawk is the default implementation, producing errors such as:
awk: /dev/fd/3: line 6: syntax error at or near ,
-- test command finished with SIGPIPE
Concretely, the match(text, pattern, variable) function does not exist
in mawk, as it cannot capture matches in variables. Work around by
substituing strings, and force a numeric comparison.
Prior to this change, each test would have, as label, it's description.
This leads to many labels, which are normally reserved for grouping
tests. Move the description into a custom property instead.
Since we rely on the JSON output for the generation of HTML reports, we
need to also extract the description from the generated JSON. However,
custom properties are only logged starting from cmake 3.30. For
back-wards compatibility, write an additional, separate environment
variable TEST_DESCRIPTION from which the description can be read as a
fallback.
For the add_timed_physim_test(), this has been already done in a parent
commit.
The unit tests have been imported with this command:
xmlstarlet sel -t
-m '//testCaseList/testCase[class="Run_Physim"]' \
-v "concat('add_timed_physim_test(physim.5g-offload.',physim_test,'.testX.Y \"',desc,'\" ',physim_test,' ',physim_run_args,')')" \
-n t2_offload_enc_nr_dlsim.xml t2_offload_dec_nr_ulsim.xml >> ../../openair1/SIMULATION/tests/CMakeLists.txt
The next commit will further cleanup this output. I commit it like this
for verification purposes.
Add two new macros that allow to register physim tests for ctest,
including timing thresholds that should be fulfilled.
First, add_timed_physim_test() registers a new test using a helper
script RunTimedTest.cmake. This is because after test execution, we need
to analyze logs, and the recommended way to do multiple steps in one
test is via a helper cmake script:
https://cmake.org/pipermail/cmake-developers/2016-February/027816.html
Second, check_physim_threshold() adds new thresholds. It takes a text to
be parsed, and a condition (e.g., "< 20") to check for the number
following the threshold (which is assumed to be present right after the
threshold). It uses a test property to count the total number of checks
(limiting them to 10), and sets environment variables for the script. I
initially planned to use a test property for checks, but those are only
valid in the same directory, and RunTimedTest.cmake seems to be assumed
by cmake to be "elsewhere", hence I needed to resort to environment
variables.
RunTimedTest.cmake is called through cmake with the test parameters and
checks. It re-constructs a list of checks [1], runs the test, and pipes
the log into a separate script that is passed all checks (see below).
Afterwards, it verifies that both the test and script passed.
A script analyze-timing.sh builds an awk script from the checks passed.
The script analyzes each line of the test output for the threshold, and
compares against the threshold. analyze-timing.sh returns success if all
checks passed.
[1] I did not manage to pass a "list" of checks in a single environment
variable through the check_physim_threshold(), which would be simpler.
Change signature of the macro to take the full test name, a separate
description (previously called a label?), and the executable and test
invocation. The overall signature becomes easier.
The rational for the full test name is that a later commit will
introduce the possibility to define physim tests that are checked for
specific times, in which case we need to reuse the test name. Thus,
using the full name from the beginning leads to more clarity, even
though they are a bit longer.
Add an additional safety check that the given simulator is actually a
target defined within the project (in other words, it actually exists).
Update the documentation correspondingly.
Most (but not all) physical simulators are defined under
openair1/SIMULATION/. Defining physical simulators below there seems to
be the more "obvious" directory anyway, as the physical simulators are
openair1/SIMULATION/tests
Phy(L1) Sim tests => PhySim (tests)
Jaroslava Fiedlerova