Add monolithic N2 handover test pipeline with B210 and Quectel
The goal of this MR is to provide an automated end-to-end N2 HO
validation with real RF (B210) and Quectel UE.
During the testing phase a post-HO assertion failure due to
uninitialized selected_plmn_identity was found: this was also fixed.
CI: Add monolithic N2 handover test pipeline (B210 + Quectel)
- New XML pipeline for N2 HO testing
- Docker Compose for two monolithic gNBs (gNB1/gNB2)
- Handover sequence: gNB1 → gNB2 → gNB1 with 5s controlled attenuation
(RC4DAT).
- Channel mapping: ch1-2 gNB1, ch3-4 gNB2
- Telnet server enabled for CI commands
- Clean-up/reset of attenuator state included
NGAP
- fix PLMN/cell location for Initial UE and Uplink NAS
- Prevents assertion in ngap_gNB_nas_uplink() due to uninitialized
selected_plmn_identity
- Initial UE Message: pass UE-selected PLMN and NR cell ID from RRC to
NGAP
- Uplink NAS Transport: use PLMN, TAC, and cell ID from the message
- Build NR CGI as gNB ID + cell ID
- Store UE-selected PLMN only in RRC UE context (for later NAS); stop
storing PLMN in NGAP UE context.
- PLMN validation when processing handover requests from the source gNB:
when PLMN validation fails, the target gNB now properly sends
NGAP_HANDOVER_FAILURE
- Store serving PLMN for later use in UE context
UE symbol based PDCCH receiver
This is a partial rebase/rewrite of !2895
The goal of this MR is to check how we can merge this.
The eventual result is stated in !2895. The way forward is to modify UE
RX procedures to work symbol by symbol by extracting the symbol loop out
of RX procedures. After all channels are modified this way, the main
thread can be modified to work symbol by symbol.
At this point only PDCCH was modified.
Fixes for T-Tracer to work with NR-UE
Fixes to align NR UE events with their definitions in T_messages.txt and
match the expected format in the tracer GUI parser.
Also fixes DL SNR to work in tracer GUI.
Support beam index in OAI 7.2 Fronthaul Interface
This branch supports beam index in OAI 7.2 Fronthaul Interface.
- For CP DL (PDSCH), nBeamIndex of the corresponding prbMapElm is set
when sending IQ data in oran_fh_if4p5_south_out. nBeamIndex of the
corresponding prbMapElm is set when sending IQ data in
oran_fh_if4p5_south_out.
- For CP UL (PUSCH and PRACH), nBeamIndex of the corresponding prbMapElm
is set in oran_fh_if4p5_south_out.
In addition, it fixes a couple of issues
- XRAN hardcodes nBeamIndex of PRACH to zero. Modify XRAN to support
setting of nBeamIndex of PRACH and update xran F release patch.
- Fix the bug in calculating fh_config->neAxc where num_beams_period is
multiplified twice
Testing methodology: Use Benetel O_RU for testing even though it does
not support beamforming. With successful registration and data transfer,
one could check the pcap on the fronthaul interface to see if CP packets
are with the right beam index.
Results:
1. Verified that CP-packets for SSB, DCI, PDSCH, PRACH, PDSCH, PUSCH,
CSI-RS and SRS are with the right beam index
2. Verified that there is no side-effect on Benetel O_RU. UE could
register properly.
3. Verified 4 SSB with ssb_PositionsInBurst_Bitmap = 85. Could see SSB
and PRACH with 4 different indices.
Doing this before using set_master_clock_rate() on the B200 seems to
invalidate synchronization setting.
[HW] USRP clock set to 1759836386.000000 sec
[...]
[HW] current pps at 3.286750, starting streaming at 4.286750
In other words, the current time setting was reset. With this change
now, we see
[HW] USRP clock set to 1759929873.000000 sec
[...]
[HW] current pps at 1759929873.000000, starting streaming at 1759929874.000000
This is because changing clock sources (I assume setting the master
clock rate impacts the clock) can result in changing the time. From the
documentation of [1]:
> Note: When changing clock sources, a previously set time will most
> likely be lost. It is recommended to set the time after changing the
> clock source. Otherwise, an unexpected time may line up with future PPS
> edges.
[1] https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html#a99254abfa5259b70a020e667eee619b9
- main.py: replace %%workspace%% with CONTAINERS.eNBSourceCodePath in Custom_Command and Custom_Script
Co-authored-by: Guido Casati <hello@guidocasati.com>
The code only checked that Msg3 would be scheduled during UL, but in
some combinations, e.g., with DSUUU, the current slot can very well be
UL as well => we would schedule DCI in UL slot.
Update cuBB image to cubb-build:25-2.1_srs-ul-heavy (release 25-2.1 with
changes in the L1 config to test heavy ul)
Co-authored-by: Reem Bahsoun <reem.bahsoun@openairinterface.org>
fix non initialized DMRS port value in case of DCI00 at UE
When transmitted PUSCH is neither scheduled by DCI format 0_1 with CRC
scrambled by C-RNTI, CS-RNTI, SP-CSI-RNTI or MCS-C-RNTI, nor
corresponding to a configured grant, the UE shall use [...] DM-RS port
0.
In UE code it was not set in case of DCI00 and that was leading to ULSCH
failures in some scenarios.
Doing this before using set_master_clock_rate() on the B200 seems to
invalidate clock settings:
[HW] USRP clock set to 1759836386.000000 sec
[...]
[HW] current pps at 3.286750, starting streaming at 4.286750
In other words, the current time setting was reset. With this change
now, we see
[HW] USRP clock set to 1759929873.000000 sec
[...]
[HW] current pps at 1759929873.000000, starting streaming at 1759929874.000000
This is because changing clock sources (I assume setting the master
clock rate impacts the clock) can result in changing the time. From the
documentation of set_time_next_pps() [1]:
> Note: When changing clock sources, a previously set time will most
> likely be lost. It is recommended to set the time after changing the
> clock source. Otherwise, an unexpected time may line up with future PPS
> edges.
[1] https://files.ettus.com/manual/classuhd_1_1usrp_1_1multi__usrp.html#a75b1c5375549e6a446d686ee7d9b4e14
The next commit introduces an "UL-heavy" configuration to be tested with
Aerial. To avoid waiting for the CN, start&stop it once at the beginning
of the CI run.
Reduce throughput requirement, as this fails once two DUs are connected
to a UE, which creates more load on the machine and frequently fails
with low throughput.
CI: Enable Physim threshold check for tests on Caracal and GH
This MR adds support for configurable timing threshold files in PhySim
tests.
- Introduces a new CMake cache variable PHYSIM_CHECK_FILES to specify
one or more timing threshold files (semicolon-separated).
- Enables optional enforcement of timing thresholds check in PhySim
tests without changing existing test definitions.
- Defines timing thresholds for LDPC encoding/decoding in nr_ulsim,
nr_dlsim and ldpctest on Caracal and GH machines.
SRS noise power
This MR implements the calculation of the noise power based on SRS in a
different way, and check whether SNR is more accurate or not. This MR is
built on top of the MR: !3460
A graph was added in T_Tracer to observe UL-SNR calculated using SRS.
See merge request description for measurement results.
For some physical simulators, we have multiple SNR iterations.
The previous version of the script correctly retained only the last
measurement, but if a previous measurement condition set RC=1, the
script would fail although the final "good" run might fulfil the
measurement condition (IOW, it would show SUCCESS, but still fail on the
check).
Instead, check if all measurement lines finish on SUCCESS, which handles
the above case.
This fixes basically two problems. First, remove "set -e", which leads
to the script exiting on error. For instance, there was the sleep which
should account for the time of the pod being in "ContainerCreating", but
might not be enough. In that case, the script would exit, leaving the
pod deployed.
Second, a fixed time is not good, as it can be too short and unduly
prolong waiting time. Replace with a loop.
Add OAIUE longrun pipeline
Add a new XML file for OAIUE longrun.
- Test description:
* 40 MHz BW, n78
* gNB with USRP N310, UE with USRP N310
* CN deployed on OC
- Test duration: 1 hour
- Test scenario: 3x DL 10 min test (20 Mbps) + 3x UL 10 min test (5
Mbps)
pods deployed through helm occasionally take more time, leading to
errors such as
Error from server (BadRequest): container "physim" in pod "oai-physims-4g-sczvl" is waiting to start: ContainerCreating
Give a bit more time before checking the logs. We cannot wait too much,
as the container will just end on FINISHED in running state, so the logs
will always be there.
Add check for cu_exists_f1_ue_data() before calling cu_get_f1_ue_data()
to prevent assertion failure when telnet command 'ci fetch_du_by_ue_id'
is used for UEs that have been handed over via N2 (and no longer have
F1 UE data).
This fixes the crash:
Assertion (ret == HASH_TABLE_OK && data != ((void *)0)) failed!
In get_hashtable_data() /oai-ran/openair2/F1AP/f1ap_ids.c:36
element for ue_id 1 not found
The function now returns NULL gracefully when F1 UE data is not available.
Co-authored-by: Guido Casati <hello@guidocasati.com>
These libraries depend on the logging module, so add the right
dependency log_headers. Indirectly, this will fulfill the condition of
T. I consider the prior usage of generate_T or {T_LIB} incorrect,
because basically none of these libraries actually, actively use T, but
rather only the logging module, which pulls in T. The dependency of
log_headers will also take care of generating and including T headers,
which is a dependency of the logging module.
For conf2uedata and related, fulfill the dependency
through conf2uedata_lib (on which the rest depends).
These libraries have no dependencies on UTIL, so it's better to omit it.
However, then the test is missing symbols for the config (which likely,
indirectly came through UTIL -> LOG -> CONFIG_LIB, so add it now onto
the test.
These libraries have an (indirect) dependency on T, either by using the
logging module (which in turn depends on T), or through other includes.
The easiest is to depend on the logging module, which will fulfil also
the T dependency.
It would equally be feasible to use the utils target (which in turn
depends on log_headers), but this would necessitate to compile, and does
not seem to be necessary.
Since E2 agent uses the config library, include it directly. This also
makes the dependency on T obsolete, because
- it is the logging headers that are needed, and
- the T dependency is fulfilled through the log_headers.
It seems that many compilation units include T.h through the config
module (as evidenced when taking out T.h from
config_load_configmodule.h). This is strange, because T.h is only used
in the logging headers. Hence, move it to there, where it is used.
coding_defs.h included defs_common.h, which pulls in many dependencies,
e.g., the logging module, which pulls in T. However, it does not use
anything of this. So remove, and include stdbool.h instead (because bool
is used in coding_defs.h.
nr_polar_defs.h relied on some includes done in coding_defs. Use
includes for pthreads and SIMD to define types that are used in
nr_polar_defs.h.
We don't use JER. This avoids compilation problems, such as:
/usr/bin/ld: openair3/NRPPA/MESSAGES/libasn1_nrppa.a(INTEGER_jer.c.o): in function `INTEGER_decode_jer':
tmp.MtrwVDhQqY/openair3/NRPPA/MESSAGES/INTEGER_jer.c:186:(.text+0x5b8): undefined reference to `jer_decode_primitive
XNAP is asn1c compiled with -no-gen-BER. Correspondingly, no
ber_decoder.h file is created. However, when running cmake followed by
ninja, it notices the absence of this file:
ninja explain: output openair2/XNAP/MESSAGES/ber_decoder.h doesn't exist
which triggers a complete rebuild of XNAP ASN.1. Without this file, the
targets are not marked as dirty.
This constant seems to be an analogon to NUMBER_OF_UE_MAX; replace with
the latter, as it does the same. (NBIoT does not work, anyway)
Fix dependencies of the NB_IoT target so it compiles.
Remove, because
1) it does not compile (the asn1c generated code should be prepend with
LTE_ etc, nobody did that)
2) a shared module makes no sense
3) the configuration is trivial to recreate
Link E2_AGENT libraries directly into nr-softmodem/lte-softmodem instead
of L2/L2_NR/F1AP. Doing that ensures that simulators, e.g.,
nr_dlsim/nr_ulsim do still compile. It also simplifies the code.
Prior to this, compiling nr_ulsim with E2_AGENT, for instance, resulted
in the following errors, fixed in this commit:
FAILED: nr_ulsim
/usr/bin/ld: openair2/E2AP/RAN_FUNCTION/libe2_ran_func_du_cucp_cuup.a(ran_func_rc.c.o): in function `send_aper_ric_ind':
/home/oai/robert/cleanup-t-dependency-cmake/openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c:643: (.text+0xee6): undefined reference to `async_event_agent_api'
/usr/bin/ld: /home/oai/robert/cleanup-t-dependency-cmake/openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c:643:(.text+0x11ca): undefined reference to `async_event_agent_api'
/usr/bin/ld: /home/oai/robert/cleanup-t-dependency-cmake/openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c:643:(.text+0x1485): undefined reference to `async_event_agent_api'
/usr/bin/ld: /home/oai/robert/cleanup-t-dependency-cmake/openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c:643:(.text+0x159c): undefined reference to `async_event_agent_api'
/usr/bin/ld: /home/oai/robert/cleanup-t-dependency-cmake/openair2/E2AP/RAN_FUNCTION/O-RAN/ran_func_rc.c:643:(.text+0x16b1): undefined reference to `async_event_agent_api'
[...]
warning: ‘genericPowerPerAntena’ defined but not used [-Wunused-function]
warning: ‘graph.text’ is used uninitialized [-Wuninitialized]
warning: ignoring return value of ‘read’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
common/utils/threadPool/measurement_display.c:55:14 warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘long long unsigned int’ [-Wformat=]
N2 handover
This MR introduces full support for N2 handover, covering handover
decision, preparation, execution, and notification between source and
target NG-RAN nodes.
Key features:
1. Handover Decision & Preparation:
- Implement N2 handover decision based on UE measurement reports.
- Generate the Handover Preparation Information message, including
RRCReconfiguration.
- Send Handover Required message from the source NG-RAN to the AMF.
2. Handover Execution:
- Decode and handle the Handover Request on the target NG-RAN.
- Process the Handover Command on the source gNB, triggering
RRCReconfiguration.
- Initiate Handover Notify after successful completion.
3 NGAP & RRC Enhancements:
- Implement NGAP encoding/decoding functions for Handover Required,
Handover Request, Handover Command, Handover Failure, and Handover
Notify (3GPP TS 38.413).
- Adds NG-RAN Status Transfer support (UL/DL) for PDCP COUNT
preservation during mobility:
- support for NGAP Uplink and Downlink RAN Status Transfer messages
(TS 38.413 §9.2.3.13-14) to enable proper PDCP COUNT value
transfer during handover or recovery: NGAP encoding/decoding for
RAN Status Transfer, PDCP helpers to extract and apply COUNT (HFN
+ SN) per DRB. This aligns with TS 38.300 §9.2.3.2.1 and TS 23.502
§4.9.1.3.3, ensuring PDCP status preservation in AM.
4. Testing:
- Introduce a Telnet command to manually trigger N2 handover using UE
ID and neighbor PCI.
- Add configuration file for testing N2 handover with a target gNB.
CI: build images on Jetson (armv8)
Add a new build pipeline for NVIDIA Jetson (armv8). Images have tag
prefix armv8_ to distinguish from gracehopper-built images prefixed arm_
(should have maybe been better with armv9_?)
Introduce a new timer for the use case where a UE should not be
scheduled anymore after a certain time. This is the case, for instance,
during handover, where the "source DU" forwards the RRCReconfiguration
and should stop scheduling this UE after some time.
Originally, transm_interrupt was introduced in 5c6d90aaff ("Use
NR_timer_t for UE transmission interrupt, e.g., RRC processing") to stop
scheduling a UE.
Commit 004403c84d ("Implement F1 UE Ctxt Modif Transmission Action
Indicator") reused this timer in an attempt to save code with an
"interrupt follow action".
In commit 89ebbb4192 ("Do not trigger ul failure in measgap scheduling
and standardize the verification if MAC is active in other scheduling
functions"), the transm_interrupt timer is used to not schedule a UE
through function nr_mac_ue_is_active(). This led to the UE not being
forwarded the RRCReconfiguration, effectively breaking handover, as the
message never reaches the UE. By using a new timer, the intended
functionality is restored.
Fixes: 89ebbb4192 ("Do not trigger ul failure in measgap scheduling
and standardize the verification if MAC is active in other
scheduling functions")
Remove this flag, as it is not used as of this commit, and introduces
confusion (see next commit: we reused the trans_interrupt timer for
the transmission action indicator stop action, where the UE should stay
active, while it is *not* when transm_interrupt is active).
For clarity, rename to native_armv9 for clarity. The check inside Python
checked for native_arm, and if we introduce native_armv8 (for jetson),
it would also match that.
Integration: `2025.w37`
* !3514 NR UE: add support for PDSCH rate matching with NZP CSI-RS
* !3630 feat(ldpc_aal): Do not try EAL init if FHI 7.2 is enabled
* !3634 reset MSG3 C-RNTI flag in case of failure
* !3566 fix 600 cppcheck notifications
* !3633 Fix PLMN print in select_amf
* !3631 Revert "Removed TRP-scheme byte from UL BF PDU to Interop with Aerial 25-1 (FAPI 10.02)"
* !3622 Populate sampled_ue_antennas
* !3601 gNB: adaptive DL/UL MCS based on reported SSB-SINR and measured PUSCH SNR
* !3637 fix bug in computing number of LCG IDs with data at UE
* !3635 Fixes and updates for UL scheduling, notably DSUUU
* !3607 Reworking BWP configuration
* !3531 Refactor measConfig
* !3612 CI: Creation of F1 Handover pipeline
* !3611 CI: Generalize PhySim tests, run PhySims on Gracehopper
Closes#985 and #975
See merge request oai/openairinterface5g!3636
CI: Generalize PhySim tests, run PhySims on Gracehopper
This changeset generalizes/harmonizes the CI infrastructure for physim
testing, and adds a new pipeline for physim tests on Gracehopper
machines.
Specifically:
- Refactor the existing CI code for running and analyzing physical
simulators. The existing code assumed those tests are deployed using
OpenShift. Instead, OpenShift/Helm-specificities (e.g., running a
"release") have been removed, and instead of hardcoding the deployment
of the physical simulators using a script for Helm, the function takes
any script that can deploy using Helm/docker/from source. The overall
appearance is the same.
- Add a new pipeline RAN-PhySim-GraceHopper-5G that executes physical
simulators on gracehopper. Reporting is done in the same way as for
existing physical simulators (e.g., like in the RAN-PhySim-Cluster-5G
pipeline), but deployed in docker instead of helm.
-> some tests don't work reliably on ARM yet and are skipped
- the RAN-gNB-N300-Timing-Phytest-LDPC pipeline has been reworked to use
the same code for testing T2 offload, with ctest, and the same
reporting.
- the CUDA LDPC test XML has been removed, as these tests don't work
reliably.
- add a new macro add_timed_physim_test() that allows to define
thresholds to be checked, so that ctest will check for them. see the
documentation for more information
CI: Creation of F1 Handover pipeline
In this MR we are modifying/adding all the required files (.xml, .conf,
.yml) in order to create a pipeline on Jenkins that allows us to test
OTA F1 handover.
Remove the runtime, which is not interesting, and add instead
- test name and if the test failed
- the threshold checks, if any, and
- possible additional information about test failures.
Prior to this change, each test would have, as label, it's description.
This leads to many labels, which are normally reserved for grouping
tests. Move the description into a custom property instead.
Since we rely on the JSON output for the generation of HTML reports, we
need to also extract the description from the generated JSON. However,
custom properties are only logged starting from cmake 3.30. For
back-wards compatibility, write an additional, separate environment
variable TEST_DESCRIPTION from which the description can be read as a
fallback.
For the add_timed_physim_test(), this has been already done in a parent
commit.
The unit tests have been imported with this command:
xmlstarlet sel -t
-m '//testCaseList/testCase[class="Run_Physim"]' \
-v "concat('add_timed_physim_test(physim.5g-offload.',physim_test,'.testX.Y \"',desc,'\" ',physim_test,' ',physim_run_args,')')" \
-n t2_offload_enc_nr_dlsim.xml t2_offload_dec_nr_ulsim.xml >> ../../openair1/SIMULATION/tests/CMakeLists.txt
The next commit will further cleanup this output. I commit it like this
for verification purposes.
Add two new macros that allow to register physim tests for ctest,
including timing thresholds that should be fulfilled.
First, add_timed_physim_test() registers a new test using a helper
script RunTimedTest.cmake. This is because after test execution, we need
to analyze logs, and the recommended way to do multiple steps in one
test is via a helper cmake script:
https://cmake.org/pipermail/cmake-developers/2016-February/027816.html
Second, check_physim_threshold() adds new thresholds. It takes a text to
be parsed, and a condition (e.g., "< 20") to check for the number
following the threshold (which is assumed to be present right after the
threshold). It uses a test property to count the total number of checks
(limiting them to 10), and sets environment variables for the script. I
initially planned to use a test property for checks, but those are only
valid in the same directory, and RunTimedTest.cmake seems to be assumed
by cmake to be "elsewhere", hence I needed to resort to environment
variables.
RunTimedTest.cmake is called through cmake with the test parameters and
checks. It re-constructs a list of checks [1], runs the test, and pipes
the log into a separate script that is passed all checks (see below).
Afterwards, it verifies that both the test and script passed.
A script analyze-timing.sh builds an awk script from the checks passed.
The script analyzes each line of the test output for the threshold, and
compares against the threshold. analyze-timing.sh returns success if all
checks passed.
[1] I did not manage to pass a "list" of checks in a single environment
variable through the check_physim_threshold(), which would be simpler.
Change signature of the macro to take the full test name, a separate
description (previously called a label?), and the executable and test
invocation. The overall signature becomes easier.
The rational for the full test name is that a later commit will
introduce the possibility to define physim tests that are checked for
specific times, in which case we need to reuse the test name. Thus,
using the full name from the beginning leads to more clarity, even
though they are a bit longer.
Add an additional safety check that the given simulator is actually a
target defined within the project (in other words, it actually exists).
Update the documentation correspondingly.
Robert Schmidt