CI: Generalize PhySim tests, run PhySims on Gracehopper
This changeset generalizes/harmonizes the CI infrastructure for physim
testing, and adds a new pipeline for physim tests on Gracehopper
machines.
Specifically:
- Refactor the existing CI code for running and analyzing physical
simulators. The existing code assumed those tests are deployed using
OpenShift. Instead, OpenShift/Helm-specificities (e.g., running a
"release") have been removed, and instead of hardcoding the deployment
of the physical simulators using a script for Helm, the function takes
any script that can deploy using Helm/docker/from source. The overall
appearance is the same.
- Add a new pipeline RAN-PhySim-GraceHopper-5G that executes physical
simulators on gracehopper. Reporting is done in the same way as for
existing physical simulators (e.g., like in the RAN-PhySim-Cluster-5G
pipeline), but deployed in docker instead of helm.
-> some tests don't work reliably on ARM yet and are skipped
- the RAN-gNB-N300-Timing-Phytest-LDPC pipeline has been reworked to use
the same code for testing T2 offload, with ctest, and the same
reporting.
- the CUDA LDPC test XML has been removed, as these tests don't work
reliably.
- add a new macro add_timed_physim_test() that allows to define
thresholds to be checked, so that ctest will check for them. see the
documentation for more information
Add two new macros that allow to register physim tests for ctest,
including timing thresholds that should be fulfilled.
First, add_timed_physim_test() registers a new test using a helper
script RunTimedTest.cmake. This is because after test execution, we need
to analyze logs, and the recommended way to do multiple steps in one
test is via a helper cmake script:
https://cmake.org/pipermail/cmake-developers/2016-February/027816.html
Second, check_physim_threshold() adds new thresholds. It takes a text to
be parsed, and a condition (e.g., "< 20") to check for the number
following the threshold (which is assumed to be present right after the
threshold). It uses a test property to count the total number of checks
(limiting them to 10), and sets environment variables for the script. I
initially planned to use a test property for checks, but those are only
valid in the same directory, and RunTimedTest.cmake seems to be assumed
by cmake to be "elsewhere", hence I needed to resort to environment
variables.
RunTimedTest.cmake is called through cmake with the test parameters and
checks. It re-constructs a list of checks [1], runs the test, and pipes
the log into a separate script that is passed all checks (see below).
Afterwards, it verifies that both the test and script passed.
A script analyze-timing.sh builds an awk script from the checks passed.
The script analyzes each line of the test output for the threshold, and
compares against the threshold. analyze-timing.sh returns success if all
checks passed.
[1] I did not manage to pass a "list" of checks in a single environment
variable through the check_physim_threshold(), which would be simpler.
Change signature of the macro to take the full test name, a separate
description (previously called a label?), and the executable and test
invocation. The overall signature becomes easier.
The rational for the full test name is that a later commit will
introduce the possibility to define physim tests that are checked for
specific times, in which case we need to reuse the test name. Thus,
using the full name from the beginning leads to more clarity, even
though they are a bit longer.
Add an additional safety check that the given simulator is actually a
target defined within the project (in other words, it actually exists).
Update the documentation correspondingly.
Most (but not all) physical simulators are defined under
openair1/SIMULATION/. Defining physical simulators below there seems to
be the more "obvious" directory anyway, as the physical simulators are
openair1/SIMULATION/tests
Phy(L1) Sim tests => PhySim (tests)
Reworking BWP configuration
This MR reworks the configuration via fail of additional BWPs. Moreover, it
allows for a single BWP to be configured for a given UE at a given time (BWP
switch possible only via reconfiguration). It also reworks PTRS configuration
for consistency with the BWP one.
Fixes and updates for UL scheduling, notably DSUUU
Fix a number of problems that stem from !3521, notably regarding Msg3
scheduling, retransmissions in "UL-heavy" TDD scenarios, and BSR
handling. Modify a CI test to run a UL-heavy TDD scenario, and add
documentation for UL-heavy TDD scenarios.
gNB: adaptive DL/UL MCS based on reported SSB-SINR and measured PUSCH SNR
Currently this is used esp. if the number of rounds is limited to 1
(e.g. disable_harq).
Revert "Removed TRP-scheme byte from UL BF PDU to Interop with Aerial 25-1 (FAPI 10.02)"
This commit broke compatibility with Aerial when compiled with
-DSCF_FAPI_10_04_SRS=ON. It was decided that this should become the
default.
Modify documentation, and use a dedicated image name to make it clear in
the CI that it has SRS.
feat(ldpc_aal): Do not try EAL init if FHI 7.2 is enabled
When FHI 7.2 is enabled, the EAL initialization is done by the FHI and
the attempt to init EAL by ldpc_aal will fail. In that case, it is not
necessary neither to provide a DPDK core list for BBDEV.
This MR offers to skip EAL init in ldpc_aal when FHI 7.2 is enabled in
order to avoid error messages in the log and to allow dropping the
useless DPDK cores argument in that case.
Also add some fixes to the LDPC offload documentation.
[FHI72 M-plane] CM improvements and PM implementation
Configuration Management:
- addition of the RU username in the fhi_72 section of the config file;
previously hardcoded to oranbenetel
- modification of the frequency offset, frame structure, fft size,
cyclic prefix, and Tx gain;
- addition of the managed delay node;
for Tx/Rx endpoints.
Performance Management implementation where OAI gNB
activates/deactivates available RU performance measurements.
Performance Management support explanation.
Example run modified according to:
- modification of the frequency offset, frame structure, fft size, cyclic prefix, and Tx gain;
- addition of the managed delay node;
for Tx/Rx endpoints.
RF emulator and improvements of the timers
RF emulator:
Add an RF library emulating a real device without the need of a real
device. This is thought to enable using phy-test without requiring a
radio device or channel simulation. The library synchronizes on the
real time clock, drops TX and generate noise on RX. Noise generation
can be disabled and the noise level configured.
Timers improvements:
- Fix standard deviation for merged timers
- Add display of ULSCH decoding timers
- Add decoding timer in AAL
Align add_boolean_option as per CMake documentation use only ON/OFF for uniformity
The add_boolean_option of CMake is not uniform everywhere. We were using
either True/False or ON/OFF. It is better to be uniform to avoid
confusion.
As an example, see here CMake documentation for an example:
https://cmake.org/cmake/help/latest/module/CMakeDependentOption.html#example-basic-usage
There are also some minor spelling corrections.
Improve MAC docs for MCS and docker for BuildKit usage
See commits for details. The first one is for fixing an omission in
!3281, the second to clarify docker build usage.
Closes: #973
CI: speed up and simplify build, remove unnecessary Python code, remove --eNB* parameter list
This cleans up the CI framework for running end-to-end tests, and
improves the integration with OAI code. Specifically
- cleanup build_oai and helpers to output all text, which is captured
better by the CI
- build some images in the Cluster-Build in parallel to increase speed
- remove all --eNB{,1,2}{IPAddress,UserName,Password} variables. Those
were indexed through "instance" and "server ID" and are replaced with
the actual node name. I consider further cleanup to introduce a
"global" XML file node name so we don't need this in each step and
instead define this once.
- individual cleanup
Refactor ULSCH scheduler towards more TDD pattern flexibility
Refactors ULSCH scheduler, notably to support "UL-heavy" TDD patterns,
e.g., DSUUU. Also allows to run TDD pattern2 with UL slots, which was a
limitation in the previous code.
Previous commits removed min_grant_mcs and instead introduced
dl/ul_min_mcs as a more generic replacement.
Fixes: 96b0cae63e ("Drop 'min_grant_mcs' configuration parameter ")
Fixes: 699afafc6f ("Add support for 'dl_min_mcs' and 'ul_min_mcs'
configuration parameters")
Libboost went from 1.74.0 to 1.83.0 alongside Ubuntu update.
GCC version is set as default to be more practical.
In Ubuntu22, apt sources were in .list files, in Ubuntu24 they are in .sources files.
Add --break-system-packages for pip installation to use same logic as before, see the pip update 23.0.1 (2023-02-17).
With Ubuntu24 liboai_device requires libpython3.12
LDPC BBDEV/AAL rework
This branch aims to generalize the existing LDPC T2 implementation to support
other DPDK BBDEV accelerators, e.g., Intel ACC100/200.
Differences from the current T2 implementation include:
1. relax CMakeLists.txt checking for custom dpdk-t2 implementation,
technically, any DPDK BBDEV should work
2. BBDEV capability checks during init
3. llr_scaling to scale the input llr accordingly for better decoding
performance on the Intel ACC100/200
4. self-managed harq buffers for the ACC200, as internal harq memory is not
available
Tested BBDEV devices:
1. Intel ACC100
2. Intel ACC200/VRB1
Rename the library to reflect that it's more a generic AAL driver than
specifically for AAL. To this end, rename some documentation.
As of this commit, the options are still called nrLDPC_coding_t2. The
next commit will correct this.
Co-authored-by: Xin Zhe Khooi <khooixz@comp.nus.edu.sg>
Add a CU-UP load tester and improve GTP performance
This adds a simple tester program for the CU-UP. It sets up a number of
UEs/bearers in the CU-UP via the E1 interface, then streams data in
downlink/uplink (as seen by the CU-UP). More information is available in
tests/nr-cuup/README.md.
Includes some fixes to properly verify that SCTP is installed, and use
"the right cmake way" to use it in cmake code (see commit for details).
Remove these F1-C port numbers. The configuration is not actually
applied, and the default F1-C port number 38472 (see F1AP_PORT_NUMBER)
is hardcoded in the F1 module.
For the moment, I leave the configuration option, in case we want to
allow to configure the port numbers later).
This is a simple CU-UP load tester. It sets up a configurable number of UEs in the
CU-UP, then sends GTP traffic in downlink/uplink in the N3-F1/F1-N3
direction at the CU-UP. It then compares the sent and received data.
Additional parameters, such as amount of traffic, test length, packet
size, and IP addresses/ports are configurable.
To run:
mkdir build && cd build && cmake .. -GNinja && ninja nr-cuup nr-cuup-load-test params_libconfig
./tests/nr-cuup/nr-cuup-load-test
./nr-cuup -O ../tests/nr-cuup/load-test.conf
The tester should be able to detect lost and reordered packets.
Example lost: Receive order 2,3,5,6 (lost packet 4)
=> on packet 2, expected 2, next=expected+1=3
=> on packet 3, expected 3, next=expected+1=4
=> on packet 5, expected 4, diff=5-4=1 (lost 1), next=payload+1=6
=> on packet 6, expected 6
=> 1 lost
Example reordered: Receiver order 2,5,3,6 (lost packet 4)
=> on packet 2, expected 2, next=expected+1=3
=> on packet 5, expected 3, diff=5-3=2 (lost 2), next=payload+1=6
=> on packet 3, expected 6, (reduce lost by 1 to 1), next=expected=6
=> on packet 6, expected 6
=> 1 lost
Example reordered: Receiver order 2,5,3,4,6 (no losses)
Since the parent commit, the ra_ResponseWindow will be chosen
automatically. Correspondingly, remove most ra_ResponseWindow options
from configuration files in the repository.
One problem in some mu=1 configs is that the ra_ResponseWindow=4 (10
slots), which might be overly restrictive if the L1 slot ahead is
somewhat close to 10 (e.g., 6), in which case we might not be always
able to allocate Msg2 in time.
A notable exception here is gnb-vnf.sa.band66.u0.25prb.nfapi.conf, in
which we still provide it (1) since it uses a non-standard configuration
of a ra_ResponseWindow longer than 10ms (see also issue #916 for more
information) and (2) as an example.
Remove --usim-test command line option
Remove the --usim-test option: to my knowledge, it's not used by users,
and is misleadingly placed in various places where there should not be a
difference w.r.t. USIM configuration.
When making the changes, I assumed that --usim-test would be 0 (the
default), which is the case in all executables when not specified in
options except for the nr_dlsim, nr_pbchsmi, nr_ulsim simulators.
The only exception is the initialization of PDCP from the MAC, which
does not seem to be correct (the PDCP is in the CU, so it does not make
sense to call it from the MAC which is in the DU).
Robert Schmidt