Extend F1 Handover test
do 1 more handover in the same test to verify robustness
- This MR keeps a simple ping after the second handover since there are
still issues due to scheduling starting too early, that is as soon as
the UE context is added. This leads to retransmissions that could
eventually lead to a test failure. The issue will be fixed in a future
MR.
add check on connected DU to verify successful handover
- a new telnet function gets in input the UE ID and returns the ID of
the connected DU
wait some time after deploying DU to mimic time for measurements
feat(oran_fhlib_5g): O-RAN 7.2 FrontHaul Interface on Arm
This MR is enabling the 7.2 FHI on Arm systems.
Even though the performance are yet very bad (UE connection is very
difficult, 100Mbps DL, almost no UL), still I would like to merge this
now since the matter of making the FHI work is rather orthogonal to the
performance issues.
So merging this feature would provide a reliable settlement on top of
which we could then progressively enhance the porting of OAI to Arm.
Especially that would make our work easier by having less features to
rebase regularly on develop.
The main features this MR is made of are:
1. Make the xran patches dual
We provide patches for both E and F releases of xran that handles x86
targets as well as Arm targets. Both patches have been tested on both
types of machines (armix, broadbill). Thanks to @brobert and @changy
for their work on xran.
2. Perform 9-bits BFP compression with ArmRAL
Since the BFP compression implementation provided with xran does rely
heavily on AVX512, we provide an alternative implementation that is
relying on Arm's RAN Acceleration Library (ArmRAL).
As of writing this commit description, only 9-bits compression is
supported. Other IQ width will be added.
This upcoming change does not prevent review to be performed since the
BFP compression functions of ArmRAL are identical except for the IQ
width. Adding further IQ width is therefore very straightforward.
3. Fixes in oran_fhlib_5g to support Arm
The library oran_fhlib_5g needed few changes to successfully compile and
run on Arm systems. Thanks to @brobert for his work on oran_fhlib_5g.
Wait before triggering a handover. First, we send it very quickly after
deployment, which can make the DU crash (should be fixed in !3332). Second, a
UE would typically first measure neighbors. While we don't do this with OAI UE,
mimic this somewhat by giving more time.
See also !3332.
* the handover is followed by a sleep and an ping test
This MR keeps a simple ping after the second handover since there are still issues due to scheduling starting too early,
that is as soon as the UE context is added. This leads to retransmissions that could eventually lead to a test failure.
The issue will be fixed in a future MR.
Temporarily deactivate DRB integrity protection in SABOX-B200 E1 testcase
to avoid CI failure due to low measured throughput. Our investigation
suggests that the low throughput is likely due to limitation of the UE
(Quectel RM500Q), which appears to restrict the maximum data rate when
integrity protection is enabled. This limitation is not observed with
Quectel RM520N.
The healthcheck interval sets when the first health check occurs after the
container starts. Increasing the healthcheck interval helps avoid marking
the container as "healthy" during the initialization phase of PNF. In
particular, USRP initialization of PNF service may take longer, and deployment
failures can occur even after the 10 seconds interval. Extending the healtcheck
interval provides more time for the service before assessing container health.
- use Ubuntu image, instead of RHEL image - gNB fails with "Illegal
instruction" during RA, when running with RHEL image
- init DPDK in VA mode
- increase number of cores Tpool
Refactor core network handling in CI, harmonize code
Prior to change set, there is a highly heterogeneous situation for core
network management in CI code: some log collection is done from groovy
files (when it could be done in python, which is easier for local
development), there are multiple commands to start a CN from XML files,
the code for starting a CN is always different (no one unified way to
start "any" CN), Ping/Iperf require some support in ci_infra.yaml, ...
To help with the above, refactor the handling of core networks in the CI
code as follows:
- define all CNs in ci_infra.yaml (basically how to deploy, undeploy,
collect logs, and on which node)
- make one Python class that reads from ci_infra.yaml and that allows to
start/stop previously defined CNs (and add unit tests...)
- make one XML command for start/stop of CNs, leveraging above class
- leverage the CNs defined in ci_infra.yaml consistently in
Ping()/Iperf()
- delete old code
Furthermore, the following supporting changes/bugfixes are done:
- repair Iperf2_Unidir() and add unit test
- fix a number of invalid escape sequence errors (there are some
remaining, which I will do soon)
- various bug fixes
- remove unused XML steps
The following cases are covered
- we have a regex whose escape sequences are not to be evaluated in the
string, but given verbatim to the regex engine. Hence, mark them as
raw strings.
- some are parameters for the cmd.run() call, but the parameters are
wrong, so remove that
- there is a `\|` that is also to be passed verbatim
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:97: SyntaxWarning: invalid escape sequence '\-'
if re.search('buildconfig.build.openshift.io/[a-zA-Z\-0-9]+ created', ret.stdout) is not None:
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:167: SyntaxWarning: invalid escape sequence '\-'
result = re.search(f'pod/(?P<pod>[a-zA-Z0-9_\-]+) created', ret.stdout)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:325: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:385: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:398: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:413: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:429: SyntaxWarning: invalid escape sequence '\|'
grep_exp = "\|".join(attemptedImages)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:431: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'for pod in $(oc get pods | tail -n +2 | awk \'{{print $1}}\'); do oc get pod $pod -o json &>> cmake_targets/log/build_pod_summary.log; done', '\$', 60)
As parent. The second change has no escape sequence, but since it's also
a regex, mark it as raw for uniformity (it doesn't hurt either).
ci-scripts/cls_module.py:165: SyntaxWarning: invalid escape sequence '\.'
result = re.search('inet (?P<ip>[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)', output.stdout)
With the server object to be passed into ping test, it's not necessary
to either hardcode an IP address or provide a template. Instead, always
fill the IP address, which simplies the Ping() code.
Verify that we don't have an IP address/hardcoded by throwing an
assertion.
This is not really used as of now, because most IPs are hardcoded. See
one of the next commits for the use of this.
To make sure that we have the correct IP, there is some code that
verifies that a hardcoded IP and the CN IP would be the same.
Note: need two separate ci_infra.yaml entries, because
- for FlexRIC, the container name is different (is the tutorial CN)
- for RFsim5G E1, we don't have ext-dn => ping UPF
- call scripts for magma MME-based deployment of OAI 5GC in
ci_infra.yaml
- implement the existing python code through scripts (which does the same)
- save the cql file in the repo, which was previously downloaded every
time
- use for porcepix-cn4g and nano-cn4g (un)deployment, ping, perf, in XML
files
- Use scripts to deploy OpenShift-based 5GC in ci_infra.yaml
- Use for oc-cn5g, oc-cn5g-20897, and oc-cn5g-20897-aerial2
for (un)deployment, ping, iperf in relevant XML files
- Convert hardcoded IP addresses to %cn_ip%
Use the core network class to configure one "end" of these tests. This
will make the tests more generic and predictable, as they are explicitly
configured through two ci_infra.yaml "entities", instead of some
implicit "EPC" IP address given in configuration.
As of this commit, no XML passes this info into the framework, so no XML
step will work. As it is many changes, the following commits will add
the necessary info one by one.
This adds a helper class to manage various core networks in the CI
testing framework. Specifically, it reads from a YAML file information
such as a node, (un-)deployment script (file), log collection
instructions, etc., in order to allow a common interface to be used to
deploy and undeploy various core network implementations.
This commit adds the class (in cls_corenetwork.py) as well as tests for
this class.
These two functions use the EPC class to potentially fill in
EPC/5GC-related information, or start a packet tracer (tshark). The only
pipeline that uses these functions is RAN-gNB-N300-Timing-Phytest-LDPC,
which does not use a core network. This functionality is not used, and
can therefore be removed.
Add a possibility to use an "infrastructure YAML file" to configure both
ends of the iperf2 call through a file.
Fix bugs in Iperf2: at least on my machine, iperf2 start time is logged
at 0.00 not 0.0, so update the regex to match both. Do not overwrite
results if the matched regex (otherwise, the result might be None at the
end) and suppress server output.
Add a unit test to test that this function actually works (on my
machine, with iperf v2.2.1), as well as a unit test for analysis.
Add support for ARM build pipeline
Modify the existing python code to be able to build images where the
image tag can be prepended with a prefix, here arm_. This is preparatory
work to create the ARM build image pipeline, and reuse the existing
internal registry on porcepix to have x86 and ARM images coexist.
Fix various bugs in the build system to allow to build on a system with
as many cores as gracehopper.
Also, fix two programs (usim, nvram) to work correctly under ARM.
After the parent commit, the registry might contain prefixed images.
This commit adds the functionality to pull such prefixed image. Since we
rename the pulled image (to have a consistent name, from whichever
registry we might pull), remove the prefix as well so that it can be
used with other CI functionality (remove image, use in test, ...)
Examples for renaming:
- pull from internal_registry for x86
internal_registry/oai-gnb:branchA => oai-ci/oai-gnb:branchA
- pull from internal registry for ARM with "arm_" prefix
internal_registry/oai-gnb:arm_branchA => oai-ci/oai-gnb:branchA
- pull from x86 other registry (e.g., openshift)
openshift/namespace/oai-gnb:branchA => oai-ci/oai-gnb:branchA
Robert Schmidt