Refactor core network handling in CI, harmonize code
Prior to change set, there is a highly heterogeneous situation for core
network management in CI code: some log collection is done from groovy
files (when it could be done in python, which is easier for local
development), there are multiple commands to start a CN from XML files,
the code for starting a CN is always different (no one unified way to
start "any" CN), Ping/Iperf require some support in ci_infra.yaml, ...
To help with the above, refactor the handling of core networks in the CI
code as follows:
- define all CNs in ci_infra.yaml (basically how to deploy, undeploy,
collect logs, and on which node)
- make one Python class that reads from ci_infra.yaml and that allows to
start/stop previously defined CNs (and add unit tests...)
- make one XML command for start/stop of CNs, leveraging above class
- leverage the CNs defined in ci_infra.yaml consistently in
Ping()/Iperf()
- delete old code
Furthermore, the following supporting changes/bugfixes are done:
- repair Iperf2_Unidir() and add unit test
- fix a number of invalid escape sequence errors (there are some
remaining, which I will do soon)
- various bug fixes
- remove unused XML steps
The following cases are covered
- we have a regex whose escape sequences are not to be evaluated in the
string, but given verbatim to the regex engine. Hence, mark them as
raw strings.
- some are parameters for the cmd.run() call, but the parameters are
wrong, so remove that
- there is a `\|` that is also to be passed verbatim
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:97: SyntaxWarning: invalid escape sequence '\-'
if re.search('buildconfig.build.openshift.io/[a-zA-Z\-0-9]+ created', ret.stdout) is not None:
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:167: SyntaxWarning: invalid escape sequence '\-'
result = re.search(f'pod/(?P<pod>[a-zA-Z0-9_\-]+) created', ret.stdout)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:325: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:385: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:398: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:413: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:429: SyntaxWarning: invalid escape sequence '\|'
grep_exp = "\|".join(attemptedImages)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:431: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'for pod in $(oc get pods | tail -n +2 | awk \'{{print $1}}\'); do oc get pod $pod -o json &>> cmake_targets/log/build_pod_summary.log; done', '\$', 60)
As parent. The second change has no escape sequence, but since it's also
a regex, mark it as raw for uniformity (it doesn't hurt either).
ci-scripts/cls_module.py:165: SyntaxWarning: invalid escape sequence '\.'
result = re.search('inet (?P<ip>[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)', output.stdout)
With the server object to be passed into ping test, it's not necessary
to either hardcode an IP address or provide a template. Instead, always
fill the IP address, which simplies the Ping() code.
Verify that we don't have an IP address/hardcoded by throwing an
assertion.
This is not really used as of now, because most IPs are hardcoded. See
one of the next commits for the use of this.
To make sure that we have the correct IP, there is some code that
verifies that a hardcoded IP and the CN IP would be the same.
Note: need two separate ci_infra.yaml entries, because
- for FlexRIC, the container name is different (is the tutorial CN)
- for RFsim5G E1, we don't have ext-dn => ping UPF
- call scripts for magma MME-based deployment of OAI 5GC in
ci_infra.yaml
- implement the existing python code through scripts (which does the same)
- save the cql file in the repo, which was previously downloaded every
time
- use for porcepix-cn4g and nano-cn4g (un)deployment, ping, perf, in XML
files
- Use scripts to deploy OpenShift-based 5GC in ci_infra.yaml
- Use for oc-cn5g, oc-cn5g-20897, and oc-cn5g-20897-aerial2
for (un)deployment, ping, iperf in relevant XML files
- Convert hardcoded IP addresses to %cn_ip%
Use the core network class to configure one "end" of these tests. This
will make the tests more generic and predictable, as they are explicitly
configured through two ci_infra.yaml "entities", instead of some
implicit "EPC" IP address given in configuration.
As of this commit, no XML passes this info into the framework, so no XML
step will work. As it is many changes, the following commits will add
the necessary info one by one.
This adds a helper class to manage various core networks in the CI
testing framework. Specifically, it reads from a YAML file information
such as a node, (un-)deployment script (file), log collection
instructions, etc., in order to allow a common interface to be used to
deploy and undeploy various core network implementations.
This commit adds the class (in cls_corenetwork.py) as well as tests for
this class.
These two functions use the EPC class to potentially fill in
EPC/5GC-related information, or start a packet tracer (tshark). The only
pipeline that uses these functions is RAN-gNB-N300-Timing-Phytest-LDPC,
which does not use a core network. This functionality is not used, and
can therefore be removed.
Add a possibility to use an "infrastructure YAML file" to configure both
ends of the iperf2 call through a file.
Fix bugs in Iperf2: at least on my machine, iperf2 start time is logged
at 0.00 not 0.0, so update the regex to match both. Do not overwrite
results if the matched regex (otherwise, the result might be None at the
end) and suppress server output.
Add a unit test to test that this function actually works (on my
machine, with iperf v2.2.1), as well as a unit test for analysis.
Add support for ARM build pipeline
Modify the existing python code to be able to build images where the
image tag can be prepended with a prefix, here arm_. This is preparatory
work to create the ARM build image pipeline, and reuse the existing
internal registry on porcepix to have x86 and ARM images coexist.
Fix various bugs in the build system to allow to build on a system with
as many cores as gracehopper.
Also, fix two programs (usim, nvram) to work correctly under ARM.
After the parent commit, the registry might contain prefixed images.
This commit adds the functionality to pull such prefixed image. Since we
rename the pulled image (to have a consistent name, from whichever
registry we might pull), remove the prefix as well so that it can be
used with other CI functionality (remove image, use in test, ...)
Examples for renaming:
- pull from internal_registry for x86
internal_registry/oai-gnb:branchA => oai-ci/oai-gnb:branchA
- pull from internal registry for ARM with "arm_" prefix
internal_registry/oai-gnb:arm_branchA => oai-ci/oai-gnb:branchA
- pull from x86 other registry (e.g., openshift)
openshift/namespace/oai-gnb:branchA => oai-ci/oai-gnb:branchA
The only currently viable way to push ARM images to our registry is to
use a custom tag prefix. Modify the python test code to specify such a
tag, and default to "" (no prefix).
After the preceding commits, it's now possible to build images on ARM.
Add a specific "native_arm" build kind to only build what we need as of
now (do not only call it "arm", as that would match the kind
"build_cross_arm64").
Add the corresponding XML. Pushing of images is disabled, as this does
not work as of now.
The (existing) Aerial pipeline on devkit uses an older version of nvipc,
which is upgraded on gracehopper. Use a glob to match them equally,
which also aligns this code with the corresponding Dockerfile, which
already uses that glob.
Local_Cmd forces use to use absolute paths for copy operations (because
that makes more clear where files are being copied). Use the absolute
path to avoid the corresponding exception.
Robert Schmidt