CI: rework log analysis to simplify and allow new use cases
The CI has some log analysis, but it is unmaintainable (huge functions
that mix eNB/gNB or lteUE/nrUE, are not tested, most of it is not used
or useful...) or limited to specific use cases (the timing-phytest time
analysis is hardcoded to source builds instead of also handling analysis
when running from docker images). This MR reimplements both functions.
For the log analysis, the CI team decided to throw away the old
analysis, and reimplement a mechanism that checks the couple of things
we care for.
- Allow to analyze phy-test timing analysis from docker files, and add a
unit test
- Rework existing log analysis to remove useless checks, and make it
more composable
* The undeployment steps (Undeploy_Object(), Terminate_eNB()) take an
<analysis> XML object which then either has a <services> subelement
with check that are whitespace delimited (similar to how other xml
task steps have <services>). For checks that contain whitespace,
they can use <service> (singular) that encodes a single check, such
that e.g., arguments could include spaces.
* Each check is of form service=check[=arguments], where service is
the docker services, check is a checking function (implemented as a
class with run(file, options) function inside cls_loganalysis.py,
and options arguments after the second = that is to be read by run()
(basically, options will contain arguments that can be freeform)
* these definitions can occur multiple times, e.g., it could be
serviceA=Check1 and serviceA=Check2=Args and it will execute both
Check1 and Check2
* if a service is declared, it will execute alse the Default check
which currently checks for an assertion
* Implemented (and inspired from the existing CI analysis) are:
- Default check to check for assertions (will be extended, to e.g.,
address/undefined behavior sanitizer analysis)
- RetxCheck: retransmission check as used in the CI
- LastLineContains to check the last line containes a specific
string
- ContainsString to check a specific string appears in logs
- EndsWithBye: check that towards the end (but not necessarily the
last line), there is a Bye. string (a workaround since we don't
always terminate with 0)
- There is a plan to use return codes after stopping in docker, but most
executables do not end with 0 most of the time, such that we needed to
abandon this plan.
- Additional fixes (see commit messages) and delete unused code
- we agreed to basically throw away the old log checker (as it is
unmaintainable)
- now: provide list of analysis to do
* all checkers in cls_log_analysis, can be unit tested (see also next
commit)
* looked up dynamically to simplify adding of new tests + config
* configured with
<analysis>
<services>DESCRIPTION [DESCRIPTION...]</services>
<service>DESCRIPTION</service>
...
</analysis>
where DESCRIPTION follows service=func[=options]:
- service is name of service to check
- func is function to call, which receives filename
- option are arbitary options to pass to func
<services> is whitespace delimited (so can take multiple service
analysis definitions)
<service> exists to allow service definitions with whitespace
changes that would not work in <services>
- I initially planned to check return code, but most softmodems actually
exit with non-zero return code, so this is still TODO
- Default analyzer (checking for assertions, ...) is always run if a
service is listed
- checks for file size
- add unit tests
Integrate GPU-Accelerated Channel Simulation + CI Test Pipeline
This MR builds on top of MR !3588 (feat: GPU Accelerated Channel
Simulation Pipeline).
The main portion of the changes is directly taken from MR !3588, and
focuses on integrating the GPU-accelerated channel simulation. On top of
that, this MR adds CI-related configurations to introduce a new channel
simulation testing pipeline into the CI --> RAN-Channel-Simulation
This MR introduces a CUDA-based pipeline to offload the
multipath_channel and add_noise channel simulation functions to the GPU.
This enables large-scale, real-time physical layer simulations not
possible on the CPU.
For full documentation, please see the file at openair1/SIMULATION/TOOLS/DOC/gpu_acceleration.md.
System & Compiler Requirements: An NVIDIA GPU with a compatible driver
and the CUDA Toolkit installed.
This feature was developed and tested on the GH200 server using the GCC
12 toolchain. To ensure reproducibility, building with GCC 12 is highly
recommended. Build Instructions
The primary flag to enable this feature is -DCUDA_ENABLE=ON. The
following command shows a sample configuration used during development
on the GH200 server:
./build_oai --phy_simulators --cmake-opt "\
-DCUDA_ENABLE=ON \
-DUSE_UNIFIED_MEMORY=ON \
-DUSE_ATS_MEMORY=OFF \
-DCMAKE_C_COMPILER=/usr/bin/gcc-12 \
-DCMAKE_CXX_COMPILER=/usr/bin/g++-12 \
-DENABLE_TESTS=ON"
Memory Model Configuration
The pipeline supports multiple CUDA memory management modes, which can
be configured at compile time. ATS Memory (-DUSE_ATS_MEMORY=ON) is
enabled by default. To use another memory model, you must first disable
ATS.
For a detailed explanation of these models, please refer to Section 3 of
the gpu_acceleration.md report.
The CI test proved not as stable as expected (because the UE triggers A3
events which we don't want), so revert this functionality while we work
on a new pipeline.
This reverts commit bc62e8abca.
- increase requirements for Event A3 trigger to avoid unwanted handovers
and allow testing reestablishment on different DUs
- increase ping_packetloss_threshold from 5 to 30 because we are forcing
a reestablishment, thus, packet loss will inevitably happen
- further adjustments to avoid A3 Events
Issue: sometimes it happens that an ul failure occurs in the handover
pipeline, UE reconnects with different UE ID and fetch_du_by_ue_id fails.
- Modified fetch_du_by_ue_id in telnetsrv_ci.c to automatically find
the connected UE when no UE ID is provided, similar to other
telnet commands like rrc_gNB_trigger_f1_ho
- Updated test cases in container_sa_f1_ho_b210_quectel.xml to use
the improved command without hardcoded UE ID parameter
- This makes the command more robust and easier to use in CI tests
We use Aerial exclusively on ARM/gracehopper, so no need to build it on
x86. On this occasion, rename the "kind" (ID which images to build) to
x86 for clarity and align with the other "kinds" (build_cross_arm,
native_armv9, ...).
CORESET start multiple of 6 fix
According to the standard any CORESET (except CSET0) would always start
from the first common resource block index which is multiple of 6 in the
BWP. This was not taken into account in the code. The BWP switch CI test
is modified to test a BWP start not multiple of 6.
An exception is the case where rb-Offset-r16 parameter is set, but can't
be used at gNB for FAPI compatibility.
Closes: #1037
This change updates the NSA-B200 CI pipeline to guarantee that the EPC is
properly terminated regardless of pipeline success or failure. This avoids
leaving EPC processes running after failed test.
PDU Session Release
This MR implements end-to-end PDU Session Release functionality
according to 3GPP TS 38.413 and TS 38.331 specifications.
Reason for Change Previous implementation had incomplete and wrongly
places tunnel cleanup during PDU session release. UP cleanup shall be
done at CUUP, RRCReconfiguration shall be sent after retrieval of
updated CellGroupConfig from the DU. This implies F1 and E1 exchanges
during PDU session release procedure.
Major Changes:
1. RRC PDU Session Release: DRBs and PDU session teardown in NGAP PDU
Session Release Response callback
- Helper Functions: Introduced rm_pduSession,
find_drb_by_pdusession_id and rm_drbs() helpers to locate and clean
up DRBs linked to a specific PDU session
- Targeted Cleanup: RRC PDU Session and per-session DRB removal in
rrc_gNB_send_NGAP_PDUSESSION_RELEASE_RESPONSE()
- Status Cleanup: Removed unused PDU_SESSION_STATUS_RELEASED status
2. RRC PDU Session Release: Refactor
rrc_gNB_process_NGAP_PDUSESSION_RELEASE_COMMAND
- Any PDU session in the pdusession_release_params list that is
present in the RRC list can be set to release, regardless of its
status
- Function Signature Update: Updated function signature and other
minor changes
- Response Handling: rrc_gNB_send_NGAP_PDUSESSION_RELEASE_RESPONSE is
sent upon completion of RRCReconfiguration
3. RRC PDU Session Release: add missing procedures E1AP Integration:
- Populates E1AP Bearer Modification for each to be released session
- PDCP Enhancement: Added nr_pdcp_release_drbs() to remove all DRBs
linked to a given PDU session in CUUP
- GTP-U Cleanup: GTP tunnel teardown handled in CUUP
e1_bearer_context_modif() using correct F1 or N3 GTP instance
4. NR PDCP/GTP:
- Enhanced Tunnel Deletion Logic: Separate handling for N3 tunnels (1
per PDU session) and F1-U tunnels (1 per DRB)
- New PDCP API Function: Added nr_pdcp_get_drb_ids_for_pdusession()
function to retrieve all DRB IDs associated with a specific PDU
session
5. RRC PDU Session Release: F1AP-based UE Context Modification:
- Replaced direct RRC message transfer with F1AP UE Context
Modification Request for proper retrieval from DU
- Cell Group Configuration: handling of updated CellGroupConfig from
DU ensures DRB release at CU
6. Docs and testing:
- Added PDU session add/remove test cases to 25 PRB RFSIM pipeline
- Documentation: Added PDU Session Release sequence diagram
- Telnet Commands: Added ci command to trigger NGAP PDU Session
Release and ciUE command to trigger establishment of a new PDU
Session
7. Additional fixes:
- UE RLC Management: ensure release of RLC entities
- Unify NG delay control before calling the RRC NGAP handler, for both
Setup and Release
8. Code Cleanup: Cleaned up unused GTP teardown functions.
Closes#867
- Enable telnet server on VNF (gNB) and UE in docker-compose.yaml
- Add new test cases for PDU session management:
* Add PDU session via UE telnet
* Release PDU session via telnet
* Verify connectivity after PDU session release
- Integrate telnet commands into existing container_5g_rfsim_u0_25prb.xml pipeline
- Test cases added before undeploy step in proper execution order
[MAC] Add UL total/used aggregated MAC stats; [E2 agent] Properly calculate
used UE PRBs in KPM; [CI] Simplify and improve the Undeploy_Object() and
implement Stop_Object()
- Create common dlul_mac_stats_t struct for both DL and UL total/used aggregated
PRBs
- Use correct unit [%] for RRU.PrbTotDl and RRU.PrbTotUl , instead of [PRBs]
- Use correct unit [Mb] for DRB.PdcpSduVolumeDL and DRB.PdcpSduVolumeUL [kb]
- Implement Stop_Object() (docker compose stop = SIGTERM)
- Simplify the Undeploy_Object() function
- Improve the logging for Deploy/Undeploy_Object() functions
- OAI-FlexRIC CI: properly stop all the containers (one by one or a subgroup of
running containers), and rename oai-flexric to nearRT-RIC container
1. Rename oai-flexric to nearRT-RIC. FlexRIC is a project combining nearRT-RIC, xApps, and E2 agent emulators.
2. Instead of undeploying all the containers at the same time, and not checking the logs for xApps and nearRT-RIC, do the following:
- Stop containers (docker compose stop) - one or subgroup of running containers in the following order:
a) xapp-kpm-moni
b) xapp-kpm-rc
c) xapp-gtp-mac-rlc-pdcp-moni
d) oai-nr-ue oai-nr-ue2
e) oai-gnb
f) nearRT-RIC
- Undeploy containers (docker compose down), collect and analyze all the logs.
PRS bug fix and CI Integration
- Fix the bug in PRS channel estimation while copying memory to compute the
impulse response
- Integrate PRS testing to CI
Some CI tasks/commands like Iperf(), Ping() get multiple nodes (via XML
parameter "nodes") to potentially run UEs on different nodes (hosts) at
the same time. However, I argue that this is not good:
- we don't actually use this -- where we specify multiple UEs, it's
always "localhost localhost..."
- it is inconsistent, as we typically have a single "node", not nodes,
and there is a possibility to harmonize (see also next commit)
- if we needed it, there would be better ways to achieve the same.
First, for hardware-based UEs, the ci_infra.yaml can specify different
UEs. For simulated UEs, it would be feasible to have multiple XML
steps run in parallel, e.g., in the XML <parallel> tag, which would
clarify that multiple UEs on different hosts run in parallel
- it reduces code.
Hence, make a single <node> for these XML steps.
CI: ensure clean iperf3 server startup
- Resolves issue with iperf3 test failure in RFSim5G multiue test
- Minor fix of iperf3 test descriptions in RFSim5G pipeline
- Add XML pipeline for N2 handover testing
- Add Docker compose for two monolithic gNBs
- Features:
* Two monolithic gNBs (gNB1, gNB2) with proper CU+DU parameters
* Gradual attenuation control using RC4DAT attenuator (5s duration)
* N2 handover sequence: gNB1 → gNB2 → gNB1
* Proper channel mapping: ch1-2 for gNB1, ch3-4 for gNB2
* Telnet server configuration for CI commands
* Handover-specific neighbor configuration
* Clean test state management with attenuator reset
By configuring the Jenkins pythonExecutor to match the test machine, we can use
localhost as the node for running individual test cases. This change makes it
easy to move tests across machines, as the only adjustment needed is updating
the pythonExecutor in Jenkins.
Jaroslava Fiedlerova