feat(oran_fhlib_5g): O-RAN 7.2 FrontHaul Interface on Arm
This MR is enabling the 7.2 FHI on Arm systems.
Even though the performance are yet very bad (UE connection is very
difficult, 100Mbps DL, almost no UL), still I would like to merge this
now since the matter of making the FHI work is rather orthogonal to the
performance issues.
So merging this feature would provide a reliable settlement on top of
which we could then progressively enhance the porting of OAI to Arm.
Especially that would make our work easier by having less features to
rebase regularly on develop.
The main features this MR is made of are:
1. Make the xran patches dual
We provide patches for both E and F releases of xran that handles x86
targets as well as Arm targets. Both patches have been tested on both
types of machines (armix, broadbill). Thanks to @brobert and @changy
for their work on xran.
2. Perform 9-bits BFP compression with ArmRAL
Since the BFP compression implementation provided with xran does rely
heavily on AVX512, we provide an alternative implementation that is
relying on Arm's RAN Acceleration Library (ArmRAL).
As of writing this commit description, only 9-bits compression is
supported. Other IQ width will be added.
This upcoming change does not prevent review to be performed since the
BFP compression functions of ArmRAL are identical except for the IQ
width. Adding further IQ width is therefore very straightforward.
3. Fixes in oran_fhlib_5g to support Arm
The library oran_fhlib_5g needed few changes to successfully compile and
run on Arm systems. Thanks to @brobert for his work on oran_fhlib_5g.
- use Ubuntu image, instead of RHEL image - gNB fails with "Illegal
instruction" during RA, when running with RHEL image
- init DPDK in VA mode
- increase number of cores Tpool
Refactor core network handling in CI, harmonize code
Prior to change set, there is a highly heterogeneous situation for core
network management in CI code: some log collection is done from groovy
files (when it could be done in python, which is easier for local
development), there are multiple commands to start a CN from XML files,
the code for starting a CN is always different (no one unified way to
start "any" CN), Ping/Iperf require some support in ci_infra.yaml, ...
To help with the above, refactor the handling of core networks in the CI
code as follows:
- define all CNs in ci_infra.yaml (basically how to deploy, undeploy,
collect logs, and on which node)
- make one Python class that reads from ci_infra.yaml and that allows to
start/stop previously defined CNs (and add unit tests...)
- make one XML command for start/stop of CNs, leveraging above class
- leverage the CNs defined in ci_infra.yaml consistently in
Ping()/Iperf()
- delete old code
Furthermore, the following supporting changes/bugfixes are done:
- repair Iperf2_Unidir() and add unit test
- fix a number of invalid escape sequence errors (there are some
remaining, which I will do soon)
- various bug fixes
- remove unused XML steps
The following cases are covered
- we have a regex whose escape sequences are not to be evaluated in the
string, but given verbatim to the regex engine. Hence, mark them as
raw strings.
- some are parameters for the cmd.run() call, but the parameters are
wrong, so remove that
- there is a `\|` that is also to be passed verbatim
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:97: SyntaxWarning: invalid escape sequence '\-'
if re.search('buildconfig.build.openshift.io/[a-zA-Z\-0-9]+ created', ret.stdout) is not None:
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:167: SyntaxWarning: invalid escape sequence '\-'
result = re.search(f'pod/(?P<pod>[a-zA-Z0-9_\-]+) created', ret.stdout)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:325: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:385: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:398: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:413: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'oc get pods.metrics.k8s.io &>> cmake_targets/log/build-metrics.log', '\$', 10)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:429: SyntaxWarning: invalid escape sequence '\|'
grep_exp = "\|".join(attemptedImages)
/home/richie/w/ci-refactor-core-handling/ci-scripts/cls_cluster.py:431: SyntaxWarning: invalid escape sequence '\$'
self.cmd.run(f'for pod in $(oc get pods | tail -n +2 | awk \'{{print $1}}\'); do oc get pod $pod -o json &>> cmake_targets/log/build_pod_summary.log; done', '\$', 60)
As parent. The second change has no escape sequence, but since it's also
a regex, mark it as raw for uniformity (it doesn't hurt either).
ci-scripts/cls_module.py:165: SyntaxWarning: invalid escape sequence '\.'
result = re.search('inet (?P<ip>[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)', output.stdout)
With the server object to be passed into ping test, it's not necessary
to either hardcode an IP address or provide a template. Instead, always
fill the IP address, which simplies the Ping() code.
Verify that we don't have an IP address/hardcoded by throwing an
assertion.
This is not really used as of now, because most IPs are hardcoded. See
one of the next commits for the use of this.
To make sure that we have the correct IP, there is some code that
verifies that a hardcoded IP and the CN IP would be the same.
Note: need two separate ci_infra.yaml entries, because
- for FlexRIC, the container name is different (is the tutorial CN)
- for RFsim5G E1, we don't have ext-dn => ping UPF
- call scripts for magma MME-based deployment of OAI 5GC in
ci_infra.yaml
- implement the existing python code through scripts (which does the same)
- save the cql file in the repo, which was previously downloaded every
time
- use for porcepix-cn4g and nano-cn4g (un)deployment, ping, perf, in XML
files
- Use scripts to deploy OpenShift-based 5GC in ci_infra.yaml
- Use for oc-cn5g, oc-cn5g-20897, and oc-cn5g-20897-aerial2
for (un)deployment, ping, iperf in relevant XML files
- Convert hardcoded IP addresses to %cn_ip%
Use the core network class to configure one "end" of these tests. This
will make the tests more generic and predictable, as they are explicitly
configured through two ci_infra.yaml "entities", instead of some
implicit "EPC" IP address given in configuration.
As of this commit, no XML passes this info into the framework, so no XML
step will work. As it is many changes, the following commits will add
the necessary info one by one.
This adds a helper class to manage various core networks in the CI
testing framework. Specifically, it reads from a YAML file information
such as a node, (un-)deployment script (file), log collection
instructions, etc., in order to allow a common interface to be used to
deploy and undeploy various core network implementations.
This commit adds the class (in cls_corenetwork.py) as well as tests for
this class.
These two functions use the EPC class to potentially fill in
EPC/5GC-related information, or start a packet tracer (tshark). The only
pipeline that uses these functions is RAN-gNB-N300-Timing-Phytest-LDPC,
which does not use a core network. This functionality is not used, and
can therefore be removed.
Add a possibility to use an "infrastructure YAML file" to configure both
ends of the iperf2 call through a file.
Fix bugs in Iperf2: at least on my machine, iperf2 start time is logged
at 0.00 not 0.0, so update the regex to match both. Do not overwrite
results if the matched regex (otherwise, the result might be None at the
end) and suppress server output.
Add a unit test to test that this function actually works (on my
machine, with iperf v2.2.1), as well as a unit test for analysis.
Add support for ARM build pipeline
Modify the existing python code to be able to build images where the
image tag can be prepended with a prefix, here arm_. This is preparatory
work to create the ARM build image pipeline, and reuse the existing
internal registry on porcepix to have x86 and ARM images coexist.
Fix various bugs in the build system to allow to build on a system with
as many cores as gracehopper.
Also, fix two programs (usim, nvram) to work correctly under ARM.
After the parent commit, the registry might contain prefixed images.
This commit adds the functionality to pull such prefixed image. Since we
rename the pulled image (to have a consistent name, from whichever
registry we might pull), remove the prefix as well so that it can be
used with other CI functionality (remove image, use in test, ...)
Examples for renaming:
- pull from internal_registry for x86
internal_registry/oai-gnb:branchA => oai-ci/oai-gnb:branchA
- pull from internal registry for ARM with "arm_" prefix
internal_registry/oai-gnb:arm_branchA => oai-ci/oai-gnb:branchA
- pull from x86 other registry (e.g., openshift)
openshift/namespace/oai-gnb:branchA => oai-ci/oai-gnb:branchA
The only currently viable way to push ARM images to our registry is to
use a custom tag prefix. Modify the python test code to specify such a
tag, and default to "" (no prefix).
After the preceding commits, it's now possible to build images on ARM.
Add a specific "native_arm" build kind to only build what we need as of
now (do not only call it "arm", as that would match the kind
"build_cross_arm64").
Add the corresponding XML. Pushing of images is disabled, as this does
not work as of now.
The (existing) Aerial pipeline on devkit uses an older version of nvipc,
which is upgraded on gracehopper. Use a glob to match them equally,
which also aligns this code with the corresponding Dockerfile, which
already uses that glob.
Robert Schmidt