For some test cases, we might want to stop a service earlier. This
functionality is done by StopObject().
Test log collection is still done by UndeployObject(), which will
collect the logs of all services (at that point, stopped or not).
Some CI tasks/commands like Iperf(), Ping() get multiple nodes (via XML
parameter "nodes") to potentially run UEs on different nodes (hosts) at
the same time. However, I argue that this is not good:
- we don't actually use this -- where we specify multiple UEs, it's
always "localhost localhost..."
- it is inconsistent, as we typically have a single "node", not nodes,
and there is a possibility to harmonize (see also next commit)
- if we needed it, there would be better ways to achieve the same.
First, for hardware-based UEs, the ci_infra.yaml can specify different
UEs. For simulated UEs, it would be feasible to have multiple XML
steps run in parallel, e.g., in the XML <parallel> tag, which would
clarify that multiple UEs on different hosts run in parallel
- it reduces code.
Hence, make a single <node> for these XML steps.
Instead of making the function a class member, make it an instance
member because (1) it is more aligned with other functions, notably,
run(), and (2) especially for RemoteCmd() it can be faster, as
successive exec_script() calls do not require a new SSH connection.
After having transitioned to direct node names in the XML, these
parameters are not relevant anymore.
The best way to view this patch is with
git show <SHA> --ignore-all-space
Pass node directly from XML to these functions
- BuildClusterImage()
- BuildImage()
- BuildProxy()
- BuildRunTests()
instead of reading the "server ID" and looking up from
eNB_IPAddress/eNB1_IPAddress/eNB2_IPAddress.
Improve the proxy build unit test by properly passing ctx information.
CI: Refactor log collection, collect also (COTS) UE logs
Refactor log handling in CI: collect according to currently used XML files. This
makes it homogeneous in the pipelines, and avoids accidental overwriting (as
happens in phytest-timing), as well as harmonizing the log placement
(e.g., no YAML lookup in Ping/Iperf).
Allow to trace and archive UE logs (e.g., Quectel via QLog, Amarisoft UE logs).
By default, no logs will be collected as of this MR, as many files are big.
Various cleanup in Jenkinsfiles, some bugfixes, add unit tests.
- Add helper function to normalize bitrate values to Mbps for calculations.
- Add helper function to extract data from iperf test report.
- Clean up the UDP analysis function, improve test report formatting.
- Fix an issue where Gbps values were incorrectly reported due to inconsistent
unit parsing.
- Update CI unit test report checks to match the new formatting.
As of the parent commit, there is a generic "may_fail" option through
which we could still ignore an error. Thus, command_fail is superfluous,
and avoids some extra code.
Further, in all these cases, we should better be informed about any
error, so let's not use may_fail for these tests.
There might be various occasions where a CI step should be attempted to
be executed, despite a chance of failing. For instance, we might want
to try executing an instable Iperf test, and still mark the CI as
passing if it fails.
This commit introduces a generic "may_fail" option that will mark any CI
test as succeeded, despite having failed.
Ideally, we would print a warning in the HTML code, but this is not
possible because each step emits its own HTML code, instead of the main
loop handling this. We can improve this in the future.
With the server object to be passed into ping test, it's not necessary
to either hardcode an IP address or provide a template. Instead, always
fill the IP address, which simplies the Ping() code.
Verify that we don't have an IP address/hardcoded by throwing an
assertion.
Use the core network class to configure one "end" of these tests. This
will make the tests more generic and predictable, as they are explicitly
configured through two ci_infra.yaml "entities", instead of some
implicit "EPC" IP address given in configuration.
As of this commit, no XML passes this info into the framework, so no XML
step will work. As it is many changes, the following commits will add
the necessary info one by one.
This adds a helper class to manage various core networks in the CI
testing framework. Specifically, it reads from a YAML file information
such as a node, (un-)deployment script (file), log collection
instructions, etc., in order to allow a common interface to be used to
deploy and undeploy various core network implementations.
This commit adds the class (in cls_corenetwork.py) as well as tests for
this class.
Add a possibility to use an "infrastructure YAML file" to configure both
ends of the iperf2 call through a file.
Fix bugs in Iperf2: at least on my machine, iperf2 start time is logged
at 0.00 not 0.0, so update the regex to match both. Do not overwrite
results if the matched regex (otherwise, the result might be None at the
end) and suppress server output.
Add a unit test to test that this function actually works (on my
machine, with iperf v2.2.1), as well as a unit test for analysis.
In CI we sometimes encouter fail of the gNB/UE deployment (and fail of
the pipeline) caused by unsuccsessful initialization of the USRP N310.
Restart the container if the health check fails during initialization.
Introduce a configurable parameter (from XML) to set max number of
attempts for the container deployment.
Store logs from failed deployment attempts.
Recent refactoring replaced the CreateWorkspace function call with a
dedicated XML step (for clarity, and because it avoids needless
recreation of workspaces). In the proxy function, the same
CreateWorkspace xml step has been added, cloning the OAI RAN repo, when
we actually need the repository for the proxy. Hence, fix this by
calling the CreateWorkspace() function manually, but only when
necessary, to build the proxy. (It would only be a problem if the
pipeline would rebuild the proxy, which did not happen because not much
work is happening on the proxy repo).
Further, refactoring and simplication of the proxy build function were
done:
- never tag proxy with develop, that makes no sense
- overwrite CreateWorkspace() info after having created and used the
workspace (the "if" to build the proxy)
- simplify copying of logs
- add simple unit test
- remove check for ubuntu; the dependcy is "docker". Simplify the code
to not obscure with a class variable.
A lot of CI code is python mixed with bash, e.g.,
ssh = getConnection(host)
ssh.run('ls')
ssh.run('echo')
At least some of this CI code would benefit if it was written in a
simple bash script, returning error codes and potentially other
information either through stdout/stderr or files, to the calling Python
code:
ssh = runScript(host, script)
# script does: ls; echo
This commit introduces the possibility to run entire scripts. The idea
is that the executor has a script (on localhost), which is either
executed locally or on a remote host. For the remote host, the script is
not copied but piped into a remotely executed bash. In both cases,
output is either returned like the Cmd.run() function with returncode
and mixed stdout/stderr, or optionally redirected into a file on the
(remote) host, which can be treated further by the Python code in later
steps.
This reimplements the CI deployment functions, and makes them a bit more
robust. Concretely:
- Introduce a generic "deployment tag" option that can be used to
override the tag to use for specific images. By default, it is the
current "branch-commitID[0:8]" tag name (requires the change from
function ImageTagToUse() to CreateTag(), impacting also pull/push
image functions)
- Avoid sed for image names, and use an .env file that docker-compose
picks up automatically; the deployment analyzes a potentially existing
.env file and updates instead of simply overriding. For instance, some
pipelines might use -asan images for the gNB and "normal" (non-asan)
images for UE, and a simple overwriting might make the -asan image
name tag disappear for the gNB, resulting in deployment failures).
Finally, undeployment removes the .env file, so that there are no
modifications in the repository after undeployment.
- Redo the previous behavior of autodetecting asan, and use that (the
current function always uses asan, no matter what)
- Remove deployKind/displayedNewTags globals, as they are not necessary
- Make the usedImage output in HTML slimmer
- On undeployment, print service names next to undeploy analysis, and
return success/failure.
- Make the functions generally less verbose and easier to read
Note that as of and only in this commit, deployment does not work, as
all the YAML files have not been updated to work with this updated
version. The next commit adds tests for the new deployment, and updates
one YAML file (also being used in the tests). The follow-up commit then
modifies all YAML files.
Use the previous commit on returning success from each task execution
function, and implement handling of errors in the main loop. Add a
script with which this is visible using Custom_Commands.
Add some basic unit tests to verify that Iperf/Ping work. Harmonize the
logging by only print in the "main" iperf/ping function, to avoid that
if we test with multiple UEs, the output intermixes. Also, do one common
mkdir before starting multiple pings/iperfs. Finally, harmonize the
paths so that Iperf and Ping log their files in the same place.
Jaroslava Fiedlerova