RAN-Container-Parent: Auto-generate artifact in failure case
RAN-Container-Parent uses Jenkinsfile-GitLab-Container, which only
triggers sub pipelines, then collects their results. However, if a sub
pipeline did not generate an artifact, the result was skipped. It would
also not show it in the automatically generated result page, when it
should.
Remedy this by automatically generating a file with a link to the
pipeline if the artifact could not be retrieved, which
1. Will give a link, which is easier for developers to follow to the
pipeline than manually clicking through Jenkins,
2. Makes the failed pipeline appear in the list of failed pipelines.
CI: test T2 with DPDK 22.11
Following the upgrade of T2 to the latest firmware, DPDK 22.11 is now required.
This MR updates the testing pipeline to ensure compatibility and deployment of
T2 physims with DPDK 22.11.
RAN-Container-Parent uses Jenkinsfile-GitLab-Container, which only
triggers sub pipelines, then collects their results. However, if a sub
pipeline did not generate an artifact, the result was skipped. It would
also not show it in the automatically generated result page, when it
should.
Remedy this by automatically generating a file with a link to the
pipeline if the artifact could not be retrieved, which
1. Will give a link, which is easier for developers to follow to the
pipeline than manually clicking through Jenkins,
2. Makes the failed pipeline appear in the list of failed pipelines.
Build and run OAI physims with DPDK version 22.11 for testing with the latest
T2 firmware release. Environment variables PKG_CONFIG_PATH and LD_LIBRARY_PATH
are set to point to the custom installation path of DPDK v22.11.
CI: Change RU in Aerial setup from Foxconn to WNC
- Change frequency in the gNB config file, from 3.75GHz to 3.5GHz
- Add high throughput tests with may_fail tag (1Gbps in DL and 90Mbps in UL)
- Update Aerial tutorial to match the changes
This commit reduces the UL throughput target in the 2x2 test to align
with the host’s performance capabilities. The adjustment ensures more
reliable and consistent CI results.
Based on observations from testing on acamas, the interval between DL
disruption and the UE synch check needs to be increased - the UE takes
longer to go out of sync . A sleep of 15 seconds has proven to be a
reliable value to ensure constant results.
Add SYS_NICE capability to enable setting thread priority and affinity
in rfsim5g tests. This improves execution speed in test cases with
heavier processing workloads.
When migrating from RHEL (on cacofonix) to Ubuntu 22.04 (on acamas), the `nc`
command needs to be replaced by `ncat`. The version of `nc` available on Ubuntu
is the OpenBSD variant, which lacks support for some options used in our scripts.
`ncat` from the Nmap is used instead.
RF simulator: Fix concurrency issues during new client connections
- Fix concurrency issues happening when a new client connects, sometimes
leading to a total softmodem deadlock.
- trace in 4G RAR processing to debug ci result
bugfix: read msg_type only after deciphering
There was a problem with accessing msg_type from the input buffer before
deciphering, giving obviously wrong value.
CI team: we should add a test in the CI with NAS ciphering + integrity
(connection to the core) set to not null (I suggest nea2/nia2). With
openair's core, this is done by setting this in the amf config file
(nia1 and nea2 at the top of the list):
supported_integrity_algorithms:
- "NIA2"
- "NIA1"
- "NIA0"
supported_encryption_algorithms:
- "NEA2"
- "NEA1"
- "NEA0"
And also put full security in the AS. This is done with this in the gnb
config file:
security = {
ciphering_algorithms = ( "nea2", "nea0" );
integrity_algorithms = ( "nia2", "nia1", "nia0" );
drb_ciphering = "yes";
drb_integrity = "yes"
};
Ideally we should test this with both openair UE and cots UE. But as a
strict minimum do for openair UE (this is where the bug was).
No need to do much traffic, a few ping is enough. But if you want to
check throughput with full security, why not.
Remove this option: some tests produce an excessive amount of output on
failure (e.g., googletest/benchmark). This seems to overwhelm the Python
CI code. Suppress this output, because we can always manually run this
test. A high-level "this test is ok/nok" should be fine for the CI.
Update/refine gNB MAC stats, update L1
Only applicable with -q option.
Print a number of statistics, e.g., rx_ulsch_sdu to track UL MAC stats
timing. Remove unused time_meas_t definitions, and fix the name for the
total gNB scheduler time.
Print the RU stats in nrL1_stats.log, as some are already there, and do
not "pollute" the periodic logs when -q is enabled.
I used this to see why there are sometimes late slots with Aerial. I
used the above stats to get a general overview. This additional patch
shows that at least in RFsim, the RLC indication in rx ulsch sdu can
take up to 20us on my machine. Given that we can have multiple PDUs in
UL (10?), this can amount to a sizeable amount of time. While this is
for another MR, I thought the cleanup would be nice to have.
Do not pollute periodic output, and rather print those into
nrL1_stats.log. A number of them are already there anyway, so transfer
the rest. Note: ru->ifdevice.tx_fhaul is omitted, as we don't have any
IF5 split in 5G.
Speed up docker deployment in CI
- Using docker compose up --wait instead of a custom healthcheck script:
from: https://docs.docker.com/reference/cli/docker/compose/up/
--wait Wait for services to be running|healthy. Implies detached mode.
--wait-timeout Maximum duration in seconds to wait for the project to be running|healthy
- Speeding up the healthchecks by reducing the interval from 1-10s to
0.5s during start period
start_period creates a special period after the start of the container
where the healthcheck is executed every start_interval . During this
time the failures do not count towards the retries counter, but
successes change the status to healthy. This allows container to reach
healthy status much sooner.
- Using init: true to speed up docker compose down/stop
`init: true` initializes the docker container with tini or
alternative: tini is used to collect all the processes and correctly
pass the termination signal to them. This allows the docker container
to be stopped without waiting 10 seconds to kill the container
- Smallest example of this is:
services:
test-init:
image: ubuntu:jammy
command: sleep 10d
Fix concurrency issues happening when a new client connects, sometimes
leading to a total softmodem deadlock.
Key changes:
- Fix sample sending on new client connection: send the first
sample only to the newly connected client.
- Remove the --rfsimulator.hanging-workaround option
- Replace global nb_ue and Sockmutex with nb_cnx and Sockmutex
in rfsimulator_state_t for better concurrency and clarity.
This modification aims to improve health status detection by using PRACH IO as
an indicator of the gNB's status (container is marked as "healthy" when the
PRACH IO value in gNB nrL1_stats.log exceeds 0.0 dB.). The new condition accounts
for scenarios where the container may not exit or produce error codes (e.g.,
“sleep” logs or "error code overflow" messages).
- `init: true` initializes the docker container with tini or alternative - it is
used to collect all the processes and correctly pass the termination signal to
them, this allows the docker container to be stopped without waiting 10 seconds
to kill the container
- speeding up the healthchecks by reducing the interval from 1-10s to 0.5s during
start period
Co-authored-by: Bartosz Podrygajlo <bartosz.podrygajlo@openairinterface.org>
CI: Upgrade Aerial version to 25-1
- Update the Nvidia L1 (cuBB) from version 24-3 to 25-1
- Decrease the number of cores dedicated to the OAI gNB; testing showed
it does not need that many
CI: Add "may_fail" step to execute flaky steps
Add a generic may_fail CI XML step option that indicates the CI step may
fail, but the CI would still be marked as succeeded (instead of skipping
all steps and mark as failed). The cause of this failure depends on the
specific step, e.g., Iperf failing because of low throughput. This
allows to clean up some code and e.g. remove the command_fail option of
Custom_Command, which is then superfluous (and less generic). An
inconvenience is that those may_fail steps will be marked as "failed" in
the HTML. Later cleanup can correct this.
The may_fail is used for executing FHI7.2 testing to push 700 Mbps in DL
(close to the maximum).
Fix some bugs and add some clean up, see commit messages.
Enabling usage of ulsyncvalidityDuration timer in SIB19
ntn-UlSyncValidityDuration in SIB19 from Spec 38.331 v17.12 A validity
duration configured by the network for assistance information which
indicates the maximum time duration (from epochTime) during which the UE
can apply assistance information without having acquired new assistance
information. The unit of ntn-UlSyncValidityDuration is second. Value s5
corresponds to 5 s, value s10 indicate 10 s and so on.
ntn-UlSyncValidityDuration-r17 ENUMERATED{ s5, s10, s15, s20, s25, s30, s35,s40, s45, s50, s55, s60, s120, s180, s240, s900}
Implementation of epochtime is under discussion.
It can be configured in configuration file according to the above
enumeration. For GEO - 240s, MEO - 20s, LEO-5s can be used.
- This is to avoid the Docker image builder from being confused in case we have multiple nvIPC files in `/opt/nvidia-ipc`
- This occurs when we have an MR to upgrade the cuBB image
"multi-ue profile" was a specific iperf option to do iperf across
multiple UEs. However, this is not necessary anymore, the Iperf
handling multiple UEs "implicitly" (based on the number of UEs).
As of the parent commit, there is a generic "may_fail" option through
which we could still ignore an error. Thus, command_fail is superfluous,
and avoids some extra code.
Further, in all these cases, we should better be informed about any
error, so let's not use may_fail for these tests.
There might be various occasions where a CI step should be attempted to
be executed, despite a chance of failing. For instance, we might want
to try executing an instable Iperf test, and still mark the CI as
passing if it fails.
This commit introduces a generic "may_fail" option that will mark any CI
test as succeeded, despite having failed.
Ideally, we would print a warning in the HTML code, but this is not
possible because each step emits its own HTML code, instead of the main
loop handling this. We can improve this in the future.
Jaroslava Fiedlerova