CI: Change RU in Aerial setup from Foxconn to WNC
- Change frequency in the gNB config file, from 3.75GHz to 3.5GHz
- Add high throughput tests with may_fail tag (1Gbps in DL and 90Mbps in UL)
- Update Aerial tutorial to match the changes
This commit reduces the UL throughput target in the 2x2 test to align
with the host’s performance capabilities. The adjustment ensures more
reliable and consistent CI results.
Based on observations from testing on acamas, the interval between DL
disruption and the UE synch check needs to be increased - the UE takes
longer to go out of sync . A sleep of 15 seconds has proven to be a
reliable value to ensure constant results.
When migrating from RHEL (on cacofonix) to Ubuntu 22.04 (on acamas), the `nc`
command needs to be replaced by `ncat`. The version of `nc` available on Ubuntu
is the OpenBSD variant, which lacks support for some options used in our scripts.
`ncat` from the Nmap is used instead.
Speed up docker deployment in CI
- Using docker compose up --wait instead of a custom healthcheck script:
from: https://docs.docker.com/reference/cli/docker/compose/up/
--wait Wait for services to be running|healthy. Implies detached mode.
--wait-timeout Maximum duration in seconds to wait for the project to be running|healthy
- Speeding up the healthchecks by reducing the interval from 1-10s to
0.5s during start period
start_period creates a special period after the start of the container
where the healthcheck is executed every start_interval . During this
time the failures do not count towards the retries counter, but
successes change the status to healthy. This allows container to reach
healthy status much sooner.
- Using init: true to speed up docker compose down/stop
`init: true` initializes the docker container with tini or
alternative: tini is used to collect all the processes and correctly
pass the termination signal to them. This allows the docker container
to be stopped without waiting 10 seconds to kill the container
- Smallest example of this is:
services:
test-init:
image: ubuntu:jammy
command: sleep 10d
CI: Add "may_fail" step to execute flaky steps
Add a generic may_fail CI XML step option that indicates the CI step may
fail, but the CI would still be marked as succeeded (instead of skipping
all steps and mark as failed). The cause of this failure depends on the
specific step, e.g., Iperf failing because of low throughput. This
allows to clean up some code and e.g. remove the command_fail option of
Custom_Command, which is then superfluous (and less generic). An
inconvenience is that those may_fail steps will be marked as "failed" in
the HTML. Later cleanup can correct this.
The may_fail is used for executing FHI7.2 testing to push 700 Mbps in DL
(close to the maximum).
Fix some bugs and add some clean up, see commit messages.
"multi-ue profile" was a specific iperf option to do iperf across
multiple UEs. However, this is not necessary anymore, the Iperf
handling multiple UEs "implicitly" (based on the number of UEs).
As of the parent commit, there is a generic "may_fail" option through
which we could still ignore an error. Thus, command_fail is superfluous,
and avoids some extra code.
Further, in all these cases, we should better be informed about any
error, so let's not use may_fail for these tests.
Step 004000 appears double in the XML description. Remove one to avoid
that it is executed twice (because the Python code goes through the list
of test descriptions and executes all that match the current ID =>
double if appears twice).
Instead of running xApps from oai-flexric container,
create dedicated xApp containers.
In addition, modify the flexric.conf: DB_PATH doesn't exist;
DB_DIR is the correct name.
Add CI test to force RRC IDLE and new connection setup with 5G-S-TMSI
This MR is a follow up to !2725 (merged)..
The primary goal is to introduce a CI test that enforces the following behavior:
- Forces the UE to enter RRC_IDLE state.
- Validates the UE's re-connection procedure using the 5G-S-TMSI as the UE
identity.
CI: Replacing Devkit server with a Grace Hopper in Aerial pipeline
Switch servers on the Aerial pipeline from Gigabyte (AMD) to Grace
Hopper (ARM), and make the necessary changes:
- Update XML, YAML, and gnb.conf
- Change the argument passed to the L1 in aerial_l1_entrypoint.sh to be
compatible with the GH
- Update the Aerial setup in TESTBench.md and Aerial_FAPI_Split_Tutorial.md docs
- Decrease the tested throughput due to an issue with the RU. Will open
another MR to increase it once the problem is fixed.
- Update the N2 and N3 IP addresses to match the Grace Hopper's in the gnb.conf.
- Remove the test XML file for pulling images in the ARM image builder.
- Update the Aerial XML file to pull ARM images.
- Decrease tested throughput (temporarily) due to problems with Foxconn RU
Extend F1 Handover test
do 1 more handover in the same test to verify robustness
- This MR keeps a simple ping after the second handover since there are
still issues due to scheduling starting too early, that is as soon as
the UE context is added. This leads to retransmissions that could
eventually lead to a test failure. The issue will be fixed in a future
MR.
add check on connected DU to verify successful handover
- a new telnet function gets in input the UE ID and returns the ID of
the connected DU
wait some time after deploying DU to mimic time for measurements
Wait before triggering a handover. First, we send it very quickly after
deployment, which can make the DU crash (should be fixed in !3332). Second, a
UE would typically first measure neighbors. While we don't do this with OAI UE,
mimic this somewhat by giving more time.
See also !3332.
* the handover is followed by a sleep and an ping test
This MR keeps a simple ping after the second handover since there are still issues due to scheduling starting too early,
that is as soon as the UE context is added. This leads to retransmissions that could eventually lead to a test failure.
The issue will be fixed in a future MR.
- use Ubuntu image, instead of RHEL image - gNB fails with "Illegal
instruction" during RA, when running with RHEL image
- init DPDK in VA mode
- increase number of cores Tpool
Refactor core network handling in CI, harmonize code
Prior to change set, there is a highly heterogeneous situation for core
network management in CI code: some log collection is done from groovy
files (when it could be done in python, which is easier for local
development), there are multiple commands to start a CN from XML files,
the code for starting a CN is always different (no one unified way to
start "any" CN), Ping/Iperf require some support in ci_infra.yaml, ...
To help with the above, refactor the handling of core networks in the CI
code as follows:
- define all CNs in ci_infra.yaml (basically how to deploy, undeploy,
collect logs, and on which node)
- make one Python class that reads from ci_infra.yaml and that allows to
start/stop previously defined CNs (and add unit tests...)
- make one XML command for start/stop of CNs, leveraging above class
- leverage the CNs defined in ci_infra.yaml consistently in
Ping()/Iperf()
- delete old code
Furthermore, the following supporting changes/bugfixes are done:
- repair Iperf2_Unidir() and add unit test
- fix a number of invalid escape sequence errors (there are some
remaining, which I will do soon)
- various bug fixes
- remove unused XML steps
With the server object to be passed into ping test, it's not necessary
to either hardcode an IP address or provide a template. Instead, always
fill the IP address, which simplies the Ping() code.
Verify that we don't have an IP address/hardcoded by throwing an
assertion.
This is not really used as of now, because most IPs are hardcoded. See
one of the next commits for the use of this.
To make sure that we have the correct IP, there is some code that
verifies that a hardcoded IP and the CN IP would be the same.
Note: need two separate ci_infra.yaml entries, because
- for FlexRIC, the container name is different (is the tutorial CN)
- for RFsim5G E1, we don't have ext-dn => ping UPF
- call scripts for magma MME-based deployment of OAI 5GC in
ci_infra.yaml
- implement the existing python code through scripts (which does the same)
- save the cql file in the repo, which was previously downloaded every
time
- use for porcepix-cn4g and nano-cn4g (un)deployment, ping, perf, in XML
files
- Use scripts to deploy OpenShift-based 5GC in ci_infra.yaml
- Use for oc-cn5g, oc-cn5g-20897, and oc-cn5g-20897-aerial2
for (un)deployment, ping, iperf in relevant XML files
- Convert hardcoded IP addresses to %cn_ip%
Robert Schmidt