Add OAIUE longrun pipeline
Add a new XML file for OAIUE longrun.
- Test description:
* 40 MHz BW, n78
* gNB with USRP N310, UE with USRP N310
* CN deployed on OC
- Test duration: 1 hour
- Test scenario: 3x DL 10 min test (20 Mbps) + 3x UL 10 min test (5
Mbps)
pods deployed through helm occasionally take more time, leading to
errors such as
Error from server (BadRequest): container "physim" in pod "oai-physims-4g-sczvl" is waiting to start: ContainerCreating
Give a bit more time before checking the logs. We cannot wait too much,
as the container will just end on FINISHED in running state, so the logs
will always be there.
Add check for cu_exists_f1_ue_data() before calling cu_get_f1_ue_data()
to prevent assertion failure when telnet command 'ci fetch_du_by_ue_id'
is used for UEs that have been handed over via N2 (and no longer have
F1 UE data).
This fixes the crash:
Assertion (ret == HASH_TABLE_OK && data != ((void *)0)) failed!
In get_hashtable_data() /oai-ran/openair2/F1AP/f1ap_ids.c:36
element for ue_id 1 not found
The function now returns NULL gracefully when F1 UE data is not available.
Co-authored-by: Guido Casati <hello@guidocasati.com>
These libraries depend on the logging module, so add the right
dependency log_headers. Indirectly, this will fulfill the condition of
T. I consider the prior usage of generate_T or {T_LIB} incorrect,
because basically none of these libraries actually, actively use T, but
rather only the logging module, which pulls in T. The dependency of
log_headers will also take care of generating and including T headers,
which is a dependency of the logging module.
For conf2uedata and related, fulfill the dependency
through conf2uedata_lib (on which the rest depends).
These libraries have no dependencies on UTIL, so it's better to omit it.
However, then the test is missing symbols for the config (which likely,
indirectly came through UTIL -> LOG -> CONFIG_LIB, so add it now onto
the test.
These libraries have an (indirect) dependency on T, either by using the
logging module (which in turn depends on T), or through other includes.
The easiest is to depend on the logging module, which will fulfil also
the T dependency.
It would equally be feasible to use the utils target (which in turn
depends on log_headers), but this would necessitate to compile, and does
not seem to be necessary.
Since E2 agent uses the config library, include it directly. This also
makes the dependency on T obsolete, because
- it is the logging headers that are needed, and
- the T dependency is fulfilled through the log_headers.
It seems that many compilation units include T.h through the config
module (as evidenced when taking out T.h from
config_load_configmodule.h). This is strange, because T.h is only used
in the logging headers. Hence, move it to there, where it is used.
coding_defs.h included defs_common.h, which pulls in many dependencies,
e.g., the logging module, which pulls in T. However, it does not use
anything of this. So remove, and include stdbool.h instead (because bool
is used in coding_defs.h.
nr_polar_defs.h relied on some includes done in coding_defs. Use
includes for pthreads and SIMD to define types that are used in
nr_polar_defs.h.
We don't use JER. This avoids compilation problems, such as:
/usr/bin/ld: openair3/NRPPA/MESSAGES/libasn1_nrppa.a(INTEGER_jer.c.o): in function `INTEGER_decode_jer':
tmp.MtrwVDhQqY/openair3/NRPPA/MESSAGES/INTEGER_jer.c:186:(.text+0x5b8): undefined reference to `jer_decode_primitive
XNAP is asn1c compiled with -no-gen-BER. Correspondingly, no
ber_decoder.h file is created. However, when running cmake followed by
ninja, it notices the absence of this file:
ninja explain: output openair2/XNAP/MESSAGES/ber_decoder.h doesn't exist
which triggers a complete rebuild of XNAP ASN.1. Without this file, the
targets are not marked as dirty.
This constant seems to be an analogon to NUMBER_OF_UE_MAX; replace with
the latter, as it does the same. (NBIoT does not work, anyway)
Fix dependencies of the NB_IoT target so it compiles.
Remove, because
1) it does not compile (the asn1c generated code should be prepend with
LTE_ etc, nobody did that)
2) a shared module makes no sense
3) the configuration is trivial to recreate
Link E2_AGENT libraries directly into nr-softmodem/lte-softmodem instead
of L2/L2_NR/F1AP. Doing that ensures that simulators, e.g.,
nr_dlsim/nr_ulsim do still compile. It also simplifies the code.
Prior to this, compiling nr_ulsim with E2_AGENT, for instance, resulted
in the following errors, fixed in this commit:
FAILED: nr_ulsim
/usr/bin/ld: openair2/E2AP/RAN_FUNCTION/libe2_ran_func_du_cucp_cuup.a(ran_func_rc.c.o): in function `send_aper_ric_ind':
/home/oai/robert/cleanup-t-dependency-cmake/openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c:643: (.text+0xee6): undefined reference to `async_event_agent_api'
/usr/bin/ld: /home/oai/robert/cleanup-t-dependency-cmake/openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c:643:(.text+0x11ca): undefined reference to `async_event_agent_api'
/usr/bin/ld: /home/oai/robert/cleanup-t-dependency-cmake/openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c:643:(.text+0x1485): undefined reference to `async_event_agent_api'
/usr/bin/ld: /home/oai/robert/cleanup-t-dependency-cmake/openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c:643:(.text+0x159c): undefined reference to `async_event_agent_api'
/usr/bin/ld: /home/oai/robert/cleanup-t-dependency-cmake/openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c:643:(.text+0x16b1): undefined reference to `async_event_agent_api'
[...]
warning: ‘genericPowerPerAntena’ defined but not used [-Wunused-function]
warning: ‘graph.text’ is used uninitialized [-Wuninitialized]
warning: ignoring return value of ‘read’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
common/utils/threadPool/measurement_display.c:55:14 warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘long long unsigned int’ [-Wformat=]
N2 handover
This MR introduces full support for N2 handover, covering handover
decision, preparation, execution, and notification between source and
target NG-RAN nodes.
Key features:
1. Handover Decision & Preparation:
- Implement N2 handover decision based on UE measurement reports.
- Generate the Handover Preparation Information message, including
RRCReconfiguration.
- Send Handover Required message from the source NG-RAN to the AMF.
2. Handover Execution:
- Decode and handle the Handover Request on the target NG-RAN.
- Process the Handover Command on the source gNB, triggering
RRCReconfiguration.
- Initiate Handover Notify after successful completion.
3 NGAP & RRC Enhancements:
- Implement NGAP encoding/decoding functions for Handover Required,
Handover Request, Handover Command, Handover Failure, and Handover
Notify (3GPP TS 38.413).
- Adds NG-RAN Status Transfer support (UL/DL) for PDCP COUNT
preservation during mobility:
- support for NGAP Uplink and Downlink RAN Status Transfer messages
(TS 38.413 §9.2.3.13-14) to enable proper PDCP COUNT value
transfer during handover or recovery: NGAP encoding/decoding for
RAN Status Transfer, PDCP helpers to extract and apply COUNT (HFN
+ SN) per DRB. This aligns with TS 38.300 §9.2.3.2.1 and TS 23.502
§4.9.1.3.3, ensuring PDCP status preservation in AM.
4. Testing:
- Introduce a Telnet command to manually trigger N2 handover using UE
ID and neighbor PCI.
- Add configuration file for testing N2 handover with a target gNB.
CI: build images on Jetson (armv8)
Add a new build pipeline for NVIDIA Jetson (armv8). Images have tag
prefix armv8_ to distinguish from gracehopper-built images prefixed arm_
(should have maybe been better with armv9_?)
Introduce a new timer for the use case where a UE should not be
scheduled anymore after a certain time. This is the case, for instance,
during handover, where the "source DU" forwards the RRCReconfiguration
and should stop scheduling this UE after some time.
Originally, transm_interrupt was introduced in 5c6d90aaff ("Use
NR_timer_t for UE transmission interrupt, e.g., RRC processing") to stop
scheduling a UE.
Commit 004403c84d ("Implement F1 UE Ctxt Modif Transmission Action
Indicator") reused this timer in an attempt to save code with an
"interrupt follow action".
In commit 89ebbb4192 ("Do not trigger ul failure in measgap scheduling
and standardize the verification if MAC is active in other scheduling
functions"), the transm_interrupt timer is used to not schedule a UE
through function nr_mac_ue_is_active(). This led to the UE not being
forwarded the RRCReconfiguration, effectively breaking handover, as the
message never reaches the UE. By using a new timer, the intended
functionality is restored.
Fixes: 89ebbb4192 ("Do not trigger ul failure in measgap scheduling
and standardize the verification if MAC is active in other
scheduling functions")
Remove this flag, as it is not used as of this commit, and introduces
confusion (see next commit: we reused the trans_interrupt timer for
the transmission action indicator stop action, where the UE should stay
active, while it is *not* when transm_interrupt is active).
For clarity, rename to native_armv9 for clarity. The check inside Python
checked for native_arm, and if we introduce native_armv8 (for jetson),
it would also match that.
Integration: `2025.w37`
* !3514 NR UE: add support for PDSCH rate matching with NZP CSI-RS
* !3630 feat(ldpc_aal): Do not try EAL init if FHI 7.2 is enabled
* !3634 reset MSG3 C-RNTI flag in case of failure
* !3566 fix 600 cppcheck notifications
* !3633 Fix PLMN print in select_amf
* !3631 Revert "Removed TRP-scheme byte from UL BF PDU to Interop with Aerial 25-1 (FAPI 10.02)"
* !3622 Populate sampled_ue_antennas
* !3601 gNB: adaptive DL/UL MCS based on reported SSB-SINR and measured PUSCH SNR
* !3637 fix bug in computing number of LCG IDs with data at UE
* !3635 Fixes and updates for UL scheduling, notably DSUUU
* !3607 Reworking BWP configuration
* !3531 Refactor measConfig
* !3612 CI: Creation of F1 Handover pipeline
* !3611 CI: Generalize PhySim tests, run PhySims on Gracehopper
Closes#985 and #975
See merge request oai/openairinterface5g!3636
CI: Generalize PhySim tests, run PhySims on Gracehopper
This changeset generalizes/harmonizes the CI infrastructure for physim
testing, and adds a new pipeline for physim tests on Gracehopper
machines.
Specifically:
- Refactor the existing CI code for running and analyzing physical
simulators. The existing code assumed those tests are deployed using
OpenShift. Instead, OpenShift/Helm-specificities (e.g., running a
"release") have been removed, and instead of hardcoding the deployment
of the physical simulators using a script for Helm, the function takes
any script that can deploy using Helm/docker/from source. The overall
appearance is the same.
- Add a new pipeline RAN-PhySim-GraceHopper-5G that executes physical
simulators on gracehopper. Reporting is done in the same way as for
existing physical simulators (e.g., like in the RAN-PhySim-Cluster-5G
pipeline), but deployed in docker instead of helm.
-> some tests don't work reliably on ARM yet and are skipped
- the RAN-gNB-N300-Timing-Phytest-LDPC pipeline has been reworked to use
the same code for testing T2 offload, with ctest, and the same
reporting.
- the CUDA LDPC test XML has been removed, as these tests don't work
reliably.
- add a new macro add_timed_physim_test() that allows to define
thresholds to be checked, so that ctest will check for them. see the
documentation for more information
CI: Creation of F1 Handover pipeline
In this MR we are modifying/adding all the required files (.xml, .conf,
.yml) in order to create a pipeline on Jenkins that allows us to test
OTA F1 handover.
Remove the runtime, which is not interesting, and add instead
- test name and if the test failed
- the threshold checks, if any, and
- possible additional information about test failures.
Prior to this change, each test would have, as label, it's description.
This leads to many labels, which are normally reserved for grouping
tests. Move the description into a custom property instead.
Since we rely on the JSON output for the generation of HTML reports, we
need to also extract the description from the generated JSON. However,
custom properties are only logged starting from cmake 3.30. For
back-wards compatibility, write an additional, separate environment
variable TEST_DESCRIPTION from which the description can be read as a
fallback.
For the add_timed_physim_test(), this has been already done in a parent
commit.
The unit tests have been imported with this command:
xmlstarlet sel -t
-m '//testCaseList/testCase[class="Run_Physim"]' \
-v "concat('add_timed_physim_test(physim.5g-offload.',physim_test,'.testX.Y \"',desc,'\" ',physim_test,' ',physim_run_args,')')" \
-n t2_offload_enc_nr_dlsim.xml t2_offload_dec_nr_ulsim.xml >> ../../openair1/SIMULATION/tests/CMakeLists.txt
The next commit will further cleanup this output. I commit it like this
for verification purposes.
Add two new macros that allow to register physim tests for ctest,
including timing thresholds that should be fulfilled.
First, add_timed_physim_test() registers a new test using a helper
script RunTimedTest.cmake. This is because after test execution, we need
to analyze logs, and the recommended way to do multiple steps in one
test is via a helper cmake script:
https://cmake.org/pipermail/cmake-developers/2016-February/027816.html
Second, check_physim_threshold() adds new thresholds. It takes a text to
be parsed, and a condition (e.g., "< 20") to check for the number
following the threshold (which is assumed to be present right after the
threshold). It uses a test property to count the total number of checks
(limiting them to 10), and sets environment variables for the script. I
initially planned to use a test property for checks, but those are only
valid in the same directory, and RunTimedTest.cmake seems to be assumed
by cmake to be "elsewhere", hence I needed to resort to environment
variables.
RunTimedTest.cmake is called through cmake with the test parameters and
checks. It re-constructs a list of checks [1], runs the test, and pipes
the log into a separate script that is passed all checks (see below).
Afterwards, it verifies that both the test and script passed.
A script analyze-timing.sh builds an awk script from the checks passed.
The script analyzes each line of the test output for the threshold, and
compares against the threshold. analyze-timing.sh returns success if all
checks passed.
[1] I did not manage to pass a "list" of checks in a single environment
variable through the check_physim_threshold(), which would be simpler.
Change signature of the macro to take the full test name, a separate
description (previously called a label?), and the executable and test
invocation. The overall signature becomes easier.
The rational for the full test name is that a later commit will
introduce the possibility to define physim tests that are checked for
specific times, in which case we need to reuse the test name. Thus,
using the full name from the beginning leads to more clarity, even
though they are a bit longer.
Add an additional safety check that the given simulator is actually a
target defined within the project (in other words, it actually exists).
Update the documentation correspondingly.
Most (but not all) physical simulators are defined under
openair1/SIMULATION/. Defining physical simulators below there seems to
be the more "obvious" directory anyway, as the physical simulators are
openair1/SIMULATION/tests
Phy(L1) Sim tests => PhySim (tests)
Refactor measConfig
- Refactor and minor improvements in get_MeasConfig function;
- Add support for band in neighbourConfiguration (It is currently
possible to handover cells with different frequencies, as long as they
are in the same band, but this MR allows the handover to also be done
between cells in different bands).
Reworking BWP configuration
This MR reworks the configuration via fail of additional BWPs. Moreover, it
allows for a single BWP to be configured for a given UE at a given time (BWP
switch possible only via reconfiguration). It also reworks PTRS configuration
for consistency with the BWP one.
Fixes and updates for UL scheduling, notably DSUUU
Fix a number of problems that stem from !3521, notably regarding Msg3
scheduling, retransmissions in "UL-heavy" TDD scenarios, and BSR
handling. Modify a CI test to run a UL-heavy TDD scenario, and add
documentation for UL-heavy TDD scenarios.
The parameter is either set to 10 (the default, so not necessary), or to
0 forcing the gNB to schedule the UE all the time, which is only
necessary in specific scenario, and here simply wrong.
In high throughput UL scenarios, at least with our configuration, the UE
only sends Short BSR. The maximum is 300000 bytes, which is completely
"scheduled away" after a couple of UL grants. Increase the periodic BSR
to 5ms to get more BSR, which results in higher UL throughput.
On BSR reception, reset sched_ul_bytes. This change means that we assume
that no data is "in flight" for this UE at time of BSR reception, or,
put differently, that we overestimate the buffer size at UE (in the
worst case).
To illustrate this, consider the following traces before/after this
change.
Before this change: Upon BSR reception, the estimated BSR of the UE is
reset, but the gNB still accounts for data "in flight". It is visible
that the calculated buffer size (estimated - scheduled) quickly goes to
zero, and we don't use all resources, despite the buffer still being
full. The maximum application throughput in this run was ~220Mbps.
[NR_MAC] SHORT BSR at 773.19, est buf 300000
[NR_MAC] ULSCH/PUSCH: 774. 6 RNTI af8e UL sched 774. 9 PRB start 0:162 TDA 2 TBS 32797 est 300000 sched 152704 est BSR 147296 TPC 1
[NR_MAC] ULSCH/PUSCH: 774. 6 RNTI af8e UL sched 774.11 PRB start 0:162 TDA 4 TBS 7172 est 300000 sched 159876 est BSR 140124 TPC 1
[NR_MAC] ULSCH/PUSCH: 774.10 RNTI af8e UL sched 774.12 PRB start 0:162 TDA 0 TBS 32797 est 292845 sched 185501 est BSR 107344 TPC 1
[NR_MAC] ULSCH/PUSCH: 774.10 RNTI af8e UL sched 774.13 PRB start 0:162 TDA 2 TBS 32797 est 292845 sched 218298 est BSR 74547 TPC 1
[NR_MAC] ULSCH/PUSCH: 774.11 RNTI af8e UL sched 774.14 PRB start 0:162 TDA 2 TBS 32797 est 260117 sched 218298 est BSR 41819 TPC 1
[NR_MAC] ULSCH/PUSCH: 774.11 RNTI af8e UL sched 774.16 PRB start 0:162 TDA 4 TBS 7172 est 260117 sched 225470 est BSR 34647 TPC 1
[NR_MAC] ULSCH/PUSCH: 774.15 RNTI af8e UL sched 774.17 PRB start 0: 35 TDA 0 TBS 7172 est 220236 sched 185501 est BSR 34735 TPC 1
[NR_MAC] ULSCH/PUSCH: 774.15 RNTI af8e UL sched 774.18 PRB start 0:162 TDA 2 TBS 32797 est 220236 sched 218298 est BSR 1938 TPC 1
[NR_MAC] ULSCH/PUSCH: 774.16 RNTI af8e UL sched 774.19 PRB start 0: 10 TDA 2 TBS 2017 est 187508 sched 187518 est BSR -10 TPC 1
[NR_MAC] SHORT BSR at 774. 9, est buf 300000
[NR_MAC] ULSCH/PUSCH: 775. 0 RNTI af8e UL sched 775. 2 PRB start 0:162 TDA 0 TBS 32797 est 260117 sched 147549 est BSR 112568 TPC 1
[NR_MAC] ULSCH/PUSCH: 775. 0 RNTI af8e UL sched 775. 3 PRB start 0:162 TDA 2 TBS 32797 est 260117 sched 180346 est BSR 79771 TPC 1
[NR_MAC] ULSCH/PUSCH: 775. 1 RNTI af8e UL sched 775. 4 PRB start 0:162 TDA 2 TBS 32797 est 227389 sched 180346 est BSR 47043 TPC 1
[NR_MAC] ULSCH/PUSCH: 775. 1 RNTI af8e UL sched 775. 6 PRB start 0:162 TDA 4 TBS 7172 est 227389 sched 187518 est BSR 39871 TPC 1
[NR_MAC] ULSCH/PUSCH: 775. 5 RNTI af8e UL sched 775. 7 PRB start 0:162 TDA 0 TBS 32797 est 180353 sched 173174 est BSR 7179 TPC 1
[NR_MAC] ULSCH/PUSCH: 775. 5 RNTI af8e UL sched 775. 8 PRB start 0: 36 TDA 2 TBS 7298 est 180353 sched 180472 est BSR -119 TPC 1
After this change: The gNB resets the number of bytes "in flight"
(sched). Thus, the calculated buffer is larger, and does not "run out"
before a new BSR arrives (which still indicates the maximum). The UE is
thus allocated all resources, but we might overestimate some resources.
The maximum application throughput in this run was ~300Mbps, limited by
MCS and not by estimated buffer size as in the previous case.
[NR_MAC] SHORT BSR at 43. 3, est buf 300000
[NR_MAC] ULSCH/PUSCH: 43.15 RNTI d3ea UL sched 43.17 PRB start 0:162 TDA 0 TBS 36897 est 263180 sched 36897 est BSR 226283 TPC 1
[NR_MAC] ULSCH/PUSCH: 43.15 RNTI d3ea UL sched 43.18 PRB start 0:162 TDA 2 TBS 36897 est 263180 sched 73794 est BSR 189386 TPC 1
[NR_MAC] ULSCH/PUSCH: 43.16 RNTI d3ea UL sched 43.19 PRB start 12:150 TDA 2 TBS 33822 est 263180 sched 107616 est BSR 155564 TPC 1
[NR_MAC] ULSCH/PUSCH: 43.16 RNTI d3ea UL sched 44. 1 PRB start 0:162 TDA 4 TBS 7941 est 263180 sched 115557 est BSR 147623 TPC 1
[NR_MAC] ULSCH/PUSCH: 44. 1 RNTI d3ea UL sched 44. 3 PRB start 0:162 TDA 0 TBS 36897 est 173698 sched 25881 est BSR 147817 TPC 1
[NR_MAC] ULSCH/PUSCH: 44. 1 RNTI d3ea UL sched 44. 4 PRB start 0:162 TDA 2 TBS 36897 est 173698 sched 62778 est BSR 110920 TPC 1
[NR_MAC] ULSCH/PUSCH: 44. 1 RNTI d3ea UL sched 44. 6 PRB start 0:162 TDA 4 TBS 7941 est 173698 sched 70719 est BSR 102979 TPC 1
[NR_MAC] SHORT BSR at 43.13, est buf 300000
[NR_MAC] ULSCH/PUSCH: 44. 5 RNTI d3ea UL sched 44. 7 PRB start 0:162 TDA 0 TBS 36897 est 292079 sched 36897 est BSR 255182 TPC 1
[NR_MAC] ULSCH/PUSCH: 44. 5 RNTI d3ea UL sched 44. 8 PRB start 0:162 TDA 2 TBS 36897 est 292079 sched 73794 est BSR 218285 TPC 1
[NR_MAC] ULSCH/PUSCH: 44. 6 RNTI d3ea UL sched 44. 9 PRB start 0:162 TDA 2 TBS 36897 est 292079 sched 110691 est BSR 181388 TPC 1
[NR_MAC] ULSCH/PUSCH: 44. 6 RNTI d3ea UL sched 44.11 PRB start 0:162 TDA 4 TBS 7941 est 292079 sched 118632 est BSR 173447 TPC 1
[NR_MAC] ULSCH/PUSCH: 44.10 RNTI d3ea UL sched 44.12 PRB start 0:162 TDA 0 TBS 36897 est 213587 sched 39972 est BSR 173615 TPC 1
[NR_MAC] ULSCH/PUSCH: 44.10 RNTI d3ea UL sched 44.13 PRB start 0:162 TDA 2 TBS 36897 est 213587 sched 76869 est BSR 136718 TPC 1
gNB: adaptive DL/UL MCS based on reported SSB-SINR and measured PUSCH SNR
Currently this is used esp. if the number of rounds is limited to 1
(e.g. disable_harq).
Revert "Removed TRP-scheme byte from UL BF PDU to Interop with Aerial 25-1 (FAPI 10.02)"
This commit broke compatibility with Aerial when compiled with
-DSCF_FAPI_10_04_SRS=ON. It was decided that this should become the
default.
Modify documentation, and use a dedicated image name to make it clear in
the CI that it has SRS.
A future commit will enable to deploy physical simulators via docker.
Hence:
- Move the script to run out of deploy_oc_physim(), to make it more
generic (later commits will introduce other scripts)
- pass the working directory (the function will be moved into
cls_oaicitest, as it fits neither cls_containerize nor cls_cluster
when deploying the tests from source)
- not using ci-scripts: don't hardcode a specific path, just use the
working directory
- avoid helm-specific pods summary, and put it into the general "physim
logs"
- remove helm-specific "release", and use a generic name for results
- always log pod logs and status to debug, even if deployment was not
successful or tests did not finish.
Instead of making the function a class member, make it an instance
member because (1) it is more aligned with other functions, notably,
run(), and (2) especially for RemoteCmd() it can be faster, as
successive exec_script() calls do not require a new SSH connection.
Robert Schmidt