Robert Schmidt
55cebe6edf
Add two new macros that allow to register physim tests for ctest, including timing thresholds that should be fulfilled. First, add_timed_physim_test() registers a new test using a helper script RunTimedTest.cmake. This is because after test execution, we need to analyze logs, and the recommended way to do multiple steps in one test is via a helper cmake script: https://cmake.org/pipermail/cmake-developers/2016-February/027816.html Second, check_physim_threshold() adds new thresholds. It takes a text to be parsed, and a condition (e.g., "< 20") to check for the number following the threshold (which is assumed to be present right after the threshold). It uses a test property to count the total number of checks (limiting them to 10), and sets environment variables for the script. I initially planned to use a test property for checks, but those are only valid in the same directory, and RunTimedTest.cmake seems to be assumed by cmake to be "elsewhere", hence I needed to resort to environment variables. RunTimedTest.cmake is called through cmake with the test parameters and checks. It re-constructs a list of checks [1], runs the test, and pipes the log into a separate script that is passed all checks (see below). Afterwards, it verifies that both the test and script passed. A script analyze-timing.sh builds an awk script from the checks passed. The script analyzes each line of the test output for the threshold, and compares against the threshold. analyze-timing.sh returns success if all checks passed. [1] I did not manage to pass a "list" of checks in a single environment variable through the check_physim_threshold(), which would be simpler.
79 lines
2.5 KiB
Bash
Executable File
79 lines
2.5 KiB
Bash
Executable File
#!/bin/bash
|
|
|
|
# This bash script builds an awk script. The awk script tries to match
|
|
# (provided) patterns and compares them against given thresholds condition
|
|
# (e.g., threshold is "< 10").
|
|
|
|
function die() { echo $@ 1>&2; exit 1; }
|
|
|
|
# RC will be the return code. If any rule fails, it will set RC=1, which will
|
|
# make the script fail. Print also every line, because the logs are piped into
|
|
# this script, but a user is typically also interested into the raw logs.
|
|
SCRIPT='
|
|
BEGIN { RC = 0; }
|
|
|
|
{ print $0 }
|
|
'
|
|
|
|
NUM=0
|
|
# for each pair of <pattern>/<condition>, add corresponding rules.
|
|
while [ $# -gt 0 ]; do
|
|
[ $# -ne 1 ] || die "unmatched <pattern>/<condition>"
|
|
PATTERN=${1}
|
|
COND=${2}
|
|
shift 2
|
|
|
|
# Add a rule that searches for a PATTERN + number, and checks against
|
|
# CONDition. If the condition does not hold, it is counted as a failure (sets
|
|
# RC to signal error). In both cases, the result is logged in an array to
|
|
# output at the end of the script.
|
|
#
|
|
# Example: pattern "PHY proc tx", condition "< 200"
|
|
# The awk script tries to match every line for "PHY proc tx _NUMBER_" (where
|
|
# number is a decimal), and compares _NUMBER_ against condition "< 200",
|
|
# i.e., "_NUMBER_ < 200".
|
|
# If the condition holds, will set "CHECK PHY proc tx _NUMBER_ < 200 SUCCESS".
|
|
# If the condition fails, will set "CHECK PHY proc tx _NUMBER_ < 200 FAIL".
|
|
SCRIPT+='
|
|
match($0, /'${PATTERN}' +([0-9]+(\.[0-9]+)?)/, n) {
|
|
if (n[1] '${COND}') {
|
|
r = "SUCCESS";
|
|
} else {
|
|
r = "FAIL";
|
|
RC = 1;
|
|
}
|
|
RESULTS['${NUM}']=sprintf("CHECK %-35s %7.2f %-8s %s", "'${PATTERN}'", n[1], " '${COND}'", r);
|
|
}
|
|
'
|
|
|
|
# Generate an additional rule for the end of the logs: if the pattern is not
|
|
# found, it is counted as a failure and logged appropriately.
|
|
# Following the above example "PHY proc tx" and "< 200": if such log is not
|
|
# found, will set "CHECK PHY proc tx < 200 NOTFOUND"
|
|
SCRIPT+='
|
|
END {
|
|
if (!RESULTS['${NUM}']) {
|
|
RESULTS['${NUM}']=sprintf("CHECK %-35s %-7s NOTFOUND", "'${PATTERN}'", "'${COND}'");
|
|
RC = 1;
|
|
}
|
|
}
|
|
'
|
|
|
|
let NUM=${NUM}+1
|
|
done
|
|
|
|
# After passing through all logs, print all results, and exit with return code
|
|
# (0 on success, i.e..all conditions checked, otherwise 1 on failure).
|
|
SCRIPT+='
|
|
END {
|
|
for (i = 0; i < '${NUM}'; ++i) {
|
|
print RESULTS[i]
|
|
}
|
|
exit RC
|
|
}
|
|
'
|
|
|
|
# Read from separate file descriptor 3: this allows us to pipe the actual log
|
|
# to analyze into awk (by using the script like so: analyze-timing.sh < log)
|
|
awk -f /dev/fd/3 3<<< ${SCRIPT}
|