Provide add_timed_physim_test() in cmake
Add two new macros that allow to register physim tests for ctest, including timing thresholds that should be fulfilled. First, add_timed_physim_test() registers a new test using a helper script RunTimedTest.cmake. This is because after test execution, we need to analyze logs, and the recommended way to do multiple steps in one test is via a helper cmake script: https://cmake.org/pipermail/cmake-developers/2016-February/027816.html Second, check_physim_threshold() adds new thresholds. It takes a text to be parsed, and a condition (e.g., "< 20") to check for the number following the threshold (which is assumed to be present right after the threshold). It uses a test property to count the total number of checks (limiting them to 10), and sets environment variables for the script. I initially planned to use a test property for checks, but those are only valid in the same directory, and RunTimedTest.cmake seems to be assumed by cmake to be "elsewhere", hence I needed to resort to environment variables. RunTimedTest.cmake is called through cmake with the test parameters and checks. It re-constructs a list of checks [1], runs the test, and pipes the log into a separate script that is passed all checks (see below). Afterwards, it verifies that both the test and script passed. A script analyze-timing.sh builds an awk script from the checks passed. The script analyzes each line of the test output for the threshold, and compares against the threshold. analyze-timing.sh returns success if all checks passed. [1] I did not manage to pass a "list" of checks in a single environment variable through the check_physim_threshold(), which would be simpler.
This commit is contained in:
@@ -0,0 +1,41 @@
|
||||
# this script expects
|
||||
# - TEST_CMD (program + options)
|
||||
# - CHECK_SCRIPT (path to analyze-timing.sh)
|
||||
# further, it analyzes the (test-specific) environment for CHECK_0, CHECK_1,
|
||||
# etc. to construct the input to the check script.
|
||||
|
||||
# the calling process writes env vars CHECK_0, CHECK_1, ...
|
||||
# each has "condition;threshold" as it's content
|
||||
# combine all necessary CHECK0, CHECK1, ... into a single array CHECKS
|
||||
# to be passed to the check script
|
||||
foreach(count RANGE 9)
|
||||
set(CHECK "CHECK_${count}")
|
||||
if (NOT DEFINED ENV{${CHECK}})
|
||||
break()
|
||||
endif()
|
||||
list(APPEND CHECKS "$ENV{${CHECK}}")
|
||||
endforeach()
|
||||
|
||||
# execute the actual test command TEST_CMD and pipe its output into the
|
||||
# check script. Afterwards, check both commands return codes.
|
||||
execute_process(COMMAND ${TEST_CMD}
|
||||
COMMAND ${CHECK_SCRIPT} ${CHECKS}
|
||||
COMMAND_ECHO STDOUT
|
||||
RESULTS_VARIABLE RET_CODES
|
||||
)
|
||||
list(LENGTH RET_CODES LEN_RET_CODES)
|
||||
if(NOT LEN_RET_CODES EQUAL 2)
|
||||
message(SEND_ERROR "execute_process() did not run both commands!")
|
||||
endif()
|
||||
|
||||
list(GET RET_CODES 0 TEST_RET_CODE)
|
||||
message(STATUS "test command finished with ${TEST_RET_CODE}")
|
||||
if(NOT ${TEST_RET_CODE} MATCHES "0")
|
||||
message(SEND_ERROR " => test failed!")
|
||||
endif()
|
||||
|
||||
list(GET RET_CODES 1 CHECK_RET_CODE)
|
||||
message(STATUS "check command finished with ${CHECK_RET_CODE}")
|
||||
if(NOT ${CHECK_RET_CODE} MATCHES "0")
|
||||
message(SEND_ERROR " => check failed!")
|
||||
endif()
|
||||
Reference in New Issue
Block a user