Robert Schmidt
55cebe6edf
Add two new macros that allow to register physim tests for ctest, including timing thresholds that should be fulfilled. First, add_timed_physim_test() registers a new test using a helper script RunTimedTest.cmake. This is because after test execution, we need to analyze logs, and the recommended way to do multiple steps in one test is via a helper cmake script: https://cmake.org/pipermail/cmake-developers/2016-February/027816.html Second, check_physim_threshold() adds new thresholds. It takes a text to be parsed, and a condition (e.g., "< 20") to check for the number following the threshold (which is assumed to be present right after the threshold). It uses a test property to count the total number of checks (limiting them to 10), and sets environment variables for the script. I initially planned to use a test property for checks, but those are only valid in the same directory, and RunTimedTest.cmake seems to be assumed by cmake to be "elsewhere", hence I needed to resort to environment variables. RunTimedTest.cmake is called through cmake with the test parameters and checks. It re-constructs a list of checks [1], runs the test, and pipes the log into a separate script that is passed all checks (see below). Afterwards, it verifies that both the test and script passed. A script analyze-timing.sh builds an awk script from the checks passed. The script analyzes each line of the test output for the threshold, and compares against the threshold. analyze-timing.sh returns success if all checks passed. [1] I did not manage to pass a "list" of checks in a single environment variable through the check_physim_threshold(), which would be simpler.
42 lines
1.4 KiB
CMake
42 lines
1.4 KiB
CMake
# this script expects
|
|
# - TEST_CMD (program + options)
|
|
# - CHECK_SCRIPT (path to analyze-timing.sh)
|
|
# further, it analyzes the (test-specific) environment for CHECK_0, CHECK_1,
|
|
# etc. to construct the input to the check script.
|
|
|
|
# the calling process writes env vars CHECK_0, CHECK_1, ...
|
|
# each has "condition;threshold" as it's content
|
|
# combine all necessary CHECK0, CHECK1, ... into a single array CHECKS
|
|
# to be passed to the check script
|
|
foreach(count RANGE 9)
|
|
set(CHECK "CHECK_${count}")
|
|
if (NOT DEFINED ENV{${CHECK}})
|
|
break()
|
|
endif()
|
|
list(APPEND CHECKS "$ENV{${CHECK}}")
|
|
endforeach()
|
|
|
|
# execute the actual test command TEST_CMD and pipe its output into the
|
|
# check script. Afterwards, check both commands return codes.
|
|
execute_process(COMMAND ${TEST_CMD}
|
|
COMMAND ${CHECK_SCRIPT} ${CHECKS}
|
|
COMMAND_ECHO STDOUT
|
|
RESULTS_VARIABLE RET_CODES
|
|
)
|
|
list(LENGTH RET_CODES LEN_RET_CODES)
|
|
if(NOT LEN_RET_CODES EQUAL 2)
|
|
message(SEND_ERROR "execute_process() did not run both commands!")
|
|
endif()
|
|
|
|
list(GET RET_CODES 0 TEST_RET_CODE)
|
|
message(STATUS "test command finished with ${TEST_RET_CODE}")
|
|
if(NOT ${TEST_RET_CODE} MATCHES "0")
|
|
message(SEND_ERROR " => test failed!")
|
|
endif()
|
|
|
|
list(GET RET_CODES 1 CHECK_RET_CODE)
|
|
message(STATUS "check command finished with ${CHECK_RET_CODE}")
|
|
if(NOT ${CHECK_RET_CODE} MATCHES "0")
|
|
message(SEND_ERROR " => check failed!")
|
|
endif()
|