This extends the functionality provided by record in that it allows
collection of data into a Clickhouse database, allowing aggregation of
data from gNB and UE for functions like channel estimation for PUSCH
transmissions that did not decode successfully at the gNB.
It is intended to be used in conjunction with Aerial Data Lake but is
not limited to it. Data access is simpler than in !3462, but could be
improved using the metadata approach from !3462.
The integration uses logfmt for fast logging, and a clickhouse DB
client, both integrated using CPM.
This change set also adds a docker-compose file for a UE on 100MHz, to
be used with Aerial configurations, and builds the tracer for Aerial and
nrUE docker files.
CI: rework log analysis to simplify and allow new use cases
The CI has some log analysis, but it is unmaintainable (huge functions
that mix eNB/gNB or lteUE/nrUE, are not tested, most of it is not used
or useful...) or limited to specific use cases (the timing-phytest time
analysis is hardcoded to source builds instead of also handling analysis
when running from docker images). This MR reimplements both functions.
For the log analysis, the CI team decided to throw away the old
analysis, and reimplement a mechanism that checks the couple of things
we care for.
- Allow to analyze phy-test timing analysis from docker files, and add a
unit test
- Rework existing log analysis to remove useless checks, and make it
more composable
* The undeployment steps (Undeploy_Object(), Terminate_eNB()) take an
<analysis> XML object which then either has a <services> subelement
with check that are whitespace delimited (similar to how other xml
task steps have <services>). For checks that contain whitespace,
they can use <service> (singular) that encodes a single check, such
that e.g., arguments could include spaces.
* Each check is of form service=check[=arguments], where service is
the docker services, check is a checking function (implemented as a
class with run(file, options) function inside cls_loganalysis.py,
and options arguments after the second = that is to be read by run()
(basically, options will contain arguments that can be freeform)
* these definitions can occur multiple times, e.g., it could be
serviceA=Check1 and serviceA=Check2=Args and it will execute both
Check1 and Check2
* if a service is declared, it will execute alse the Default check
which currently checks for an assertion
* Implemented (and inspired from the existing CI analysis) are:
- Default check to check for assertions (will be extended, to e.g.,
address/undefined behavior sanitizer analysis)
- RetxCheck: retransmission check as used in the CI
- LastLineContains to check the last line containes a specific
string
- ContainsString to check a specific string appears in logs
- EndsWithBye: check that towards the end (but not necessarily the
last line), there is a Bye. string (a workaround since we don't
always terminate with 0)
- There is a plan to use return codes after stopping in docker, but most
executables do not end with 0 most of the time, such that we needed to
abandon this plan.
- Additional fixes (see commit messages) and delete unused code
GPU Channel Simulation: nr_dlsim/nr_ulsim Integration + CI Test
This MR is a follow-up to !3886 (merged) and extends CUDA-based channel
simulation support to nr_dlsim and nr_ulsim, together with the
corresponding CI tests.
The MR incorporates:
- Cherry-picked commits e0c204f7 and 5f344876 from !3588
- Integration of new CI physim tests using GPU-accelerated channel
simulation in nr_dlsim and nr_ulsim tests, see commit c727b3bb
New CI job: RAN-Channel-Simulation
https://jenkins-oai.eurecom.fr/job/RAN-Channel-Simulation/
[CI] Pull Ubuntu base image from internal registry
This MR updates the Docker build process for OAI images to use a private
registry for the Ubuntu base image ubuntu:noble. This will reduce the
possibility for CI disruption if no internet connection exists.
- Replace FROM ubuntu:noble with FROM <DEFAULT_REGISTRY>/ubuntu:noble in Ubuntu Dockerfiles
This change overcomes the pull rate limit issues from DockerHub in the
CI.
The <DEFAULT_REGISTRY> contains a multi-architecture image (arm64 and
amd64)
$ docker buildx imagetools inspect gracehopper3-oai.sboai.cs.eurecom.fr/ubuntu:noble | grep Platform
Platform: linux/amd64
Platform: linux/arm64
- The scripts run from the jenkins server everyday using a crontab.
- It pulls the AMD and ARM ubuntu:noble image on respective servers and
checks if the internal registry is in sync.
- If not, it pushes the ubuntu:noble image to the registry to remain in
sync with dockerhub.
It was initially foreseen to just check the program exit code, but most
executables don't return with 0 (so the CI would always fail). Add a
separate check that will check for the Bye., only going through the last
X lines instead of the entire file.
- we agreed to basically throw away the old log checker (as it is
unmaintainable)
- now: provide list of analysis to do
* all checkers in cls_log_analysis, can be unit tested (see also next
commit)
* looked up dynamically to simplify adding of new tests + config
* configured with
<analysis>
<services>DESCRIPTION [DESCRIPTION...]</services>
<service>DESCRIPTION</service>
...
</analysis>
where DESCRIPTION follows service=func[=options]:
- service is name of service to check
- func is function to call, which receives filename
- option are arbitary options to pass to func
<services> is whitespace delimited (so can take multiple service
analysis definitions)
<service> exists to allow service definitions with whitespace
changes that would not work in <services>
- I initially planned to check return code, but most softmodems actually
exit with non-zero return code, so this is still TODO
- Default analyzer (checking for assertions, ...) is always run if a
service is listed
- checks for file size
- add unit tests
- it is spaghetti code: difficult to understand which code paths are
taken
- the only effect seems to be in the UE log analysis function: it might
make the log fail. However, we test that iperf is successful
(Iperf2_Unidir()), and would detect before the log analysis if MBMS
traffic does not work
- the rest is only messages that likely nobody reads
The real reason for this change is to reduce the number of class members
in the ran.py class, to simplify the functioning of the eNB analysis
function (no self) to reduce dependencies on the ran.py class.
set -e makes the script exit on error. However, there is code to handle
a non-zero return code of the script, and -e prevents from writing the
footer.
add set -x to see all the commands that are being executed.
Use a raw string as suggested:
ci-scripts/cls_static_code_analysis.py:93: SyntaxWarning: invalid escape sequence '\:'
result = re.search('([a-zA-Z0-9\:\-\.\/])+\.git', self.ranRepository)
ci-scripts/cls_static_code_analysis.py:125: SyntaxWarning: invalid escape sequence '\.'
ret = re.search('cppcheck version="(?P<version>[0-9\.]+)"', str(line))
ci-scripts/cls_oai_html.py:221: SyntaxWarning: invalid escape sequence '\/'
cmd = "sed -i -e 's/__STATE_" + self.htmlTabNames[0] + "__/<span class=\"glyphicon glyphicon-remove\"><\/span>/' test_results.html"
ci-scripts/ran.py:428: SyntaxWarning: "\[" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\["? A raw string is also an option.
result = re.search('\[gNB [0-9]+\]\[RAPROC\] PUSCH with TC_RNTI 0x[0-9a-fA-F]+ received correctly, adding UE MAC Context RNTI 0x[0-9a-fA-F]+', str(line))
ci-scripts/ran.py:457: SyntaxWarning: "\[" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\["? A raw string is also an option.
result = re.search('\[PHY\]\s+problem receiving samples', str(line))
ci-scripts/ran.py:461: SyntaxWarning: "\[" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\["? A raw string is also an option.
result = re.search('\[MAC\]\s+Removing UE', str(line))
I am not sure we ever need the millisecond output. The date is typically
also clear from the output (we could print the date at the beginning,
once, instead, but this is not done here. Finally, in Jenkins, we
typically get time stamps. Thus, it's not necessary to print this by
default.
On the other hand, since it might be useful when running locally, make
time stamps in logs configurable.
Since the grand-parent commit, if possible, the gNBs synchronize on a
slot-level boundary. Some configuration files/command line overrides
left the same SSB position for two DUs in RFsim-based handover
scenarios, which then makes that the UE cannot decode the SSB.
This commit changes the positions such that this is possible again. In
the F1 HO docker-compose
(ci-scripts/yaml_files/5g_f1_rfsimulator/docker-compose.yaml), we set it
to 2 to send only one SSB (not 2 with bitmap 3).
fix(ci): remove git reset command from Jenkinsfile
- By default, the variable eNB_CommitID is set as 'develop' in the
config
- When OAI-FLEXRIC-RAN-Integration-Test is triggered by upstream FlexRIC
push events, it takes develop
- Triggered by RAN takes commit SHA where origin/SHA does not work
Below are issues seen after this week's merge to develop:
https://jenkins-oai.eurecom.fr/view/RAN/job/RAN-Ubuntu-ARM-Image-Builder/4103/pipeline-overview/?selected-node=44
08:43:07 + git reset --hard origin/9777d23173a740de7cf982659fae13ad6627d3aa
08:43:07 fatal: ambiguous argument 'origin/9777d23173a740de7cf982659fae13ad6627d3aa': unknown revision or path not in the working tree.
When triggered through upstream FlexRIC (as a PUSH Event), it is taking
the branch as shown in the logs:
https://jenkins-oai.eurecom.fr/job/OAI-FLEXRIC-RAN-Integration-Test/3832/pipeline-overview/?selected-node=47
13:45:27 + git reset --hard origin/develop
13:45:27 HEAD is now at 9777d23173 Merge branch 'integration_2026_w06' into 'develop'
CI: Rename RAN-SA-2x2-Module-CN5G pipeline
RAN-SA-2x2-Module-CN5G performs tests not only with 2x2 but also with a
4x4 MIMO configuration. The pipeline name has been generalized to
RAN-SA-Multi-Antenna-CN5G to cover multiple antenna configurations.
RRC QoS refactor: PDU Session Modify support, SDAP fixes, and code improvements
The goal of this MR is to prepare the RRC/NGAP layer for the multi-QoS
support. The MR:
- enables PDU Session Resource Modify support in RRC/NGAP/E1AP
- fixes SDAP header mapping
- enables SDAP by default
- refactors QoS flow handling in RRC/NGAP
- improves error handling, and code maintainability in RRC/NGAP
Changes:
- NGAP PDU Session Modify:
* add NGAP PDU Session Resource Modify Request/Response handling with
proper error reporting for unrecognized sessions and AMF UE NGAP ID
validation
- E1AP Bearer Context Modification:
* add DRB To Remove List IE to support DRB release during modify
* rename numQosFlow2Setup to numQosFlowsMod in DRB To Modify List
- RRC QoS flow handling:
* enable multi-QoS flow support per DRB
* add utilities to map NGAP QoS parameters to E1AP
* filter QoS flows by DRB ID
- SDAP configuration:
* Fix SDAP header UL/DL mapping between internal representation and
ASN.1 enum (invert mapping)
* change SDAP header fields from long to bool for clarity
- F1 DRB setup refactoring:
* Extract shared helpers for F1 DRB setup list filling
* reduce duplication between E1 response and direct DRB setup paths
- AMBR management:
* Store AMBR in UE context
* validate AMBR is set before bearer setup
- Configuration:
* remove unused drbs configuration parameter
* enable SDAP by default (change default from false to true)
RAN-SA-2x2-Module-CN5G performs tests not only with 2x2 but also with a 4x4 MIMO
configuration. The pipeline name has been generalized to RAN-SA-Multi-Antenna-CN5G
to cover multiple antenna configurations.
Integrate GPU-Accelerated Channel Simulation + CI Test Pipeline
This MR builds on top of MR !3588 (feat: GPU Accelerated Channel
Simulation Pipeline).
The main portion of the changes is directly taken from MR !3588, and
focuses on integrating the GPU-accelerated channel simulation. On top of
that, this MR adds CI-related configurations to introduce a new channel
simulation testing pipeline into the CI --> RAN-Channel-Simulation
This MR introduces a CUDA-based pipeline to offload the
multipath_channel and add_noise channel simulation functions to the GPU.
This enables large-scale, real-time physical layer simulations not
possible on the CPU.
For full documentation, please see the file at openair1/SIMULATION/TOOLS/DOC/gpu_acceleration.md.
System & Compiler Requirements: An NVIDIA GPU with a compatible driver
and the CUDA Toolkit installed.
This feature was developed and tested on the GH200 server using the GCC
12 toolchain. To ensure reproducibility, building with GCC 12 is highly
recommended. Build Instructions
The primary flag to enable this feature is -DCUDA_ENABLE=ON. The
following command shows a sample configuration used during development
on the GH200 server:
./build_oai --phy_simulators --cmake-opt "\
-DCUDA_ENABLE=ON \
-DUSE_UNIFIED_MEMORY=ON \
-DUSE_ATS_MEMORY=OFF \
-DCMAKE_C_COMPILER=/usr/bin/gcc-12 \
-DCMAKE_CXX_COMPILER=/usr/bin/g++-12 \
-DENABLE_TESTS=ON"
Memory Model Configuration
The pipeline supports multiple CUDA memory management modes, which can
be configured at compile time. ATS Memory (-DUSE_ATS_MEMORY=ON) is
enabled by default. To use another memory model, you must first disable
ATS.
For a detailed explanation of these models, please refer to Section 3 of
the gpu_acceleration.md report.
- Change gNB RRC enable_sdap default to true
- Update RRC and RUNMODEM docs to reflect new default behavior
- Clean up CI gNB configs by removing redundant enable_sdap=1
- Keep explicit SDAP disable config for NFAPI gNB VNF
Jaroslava Fiedlerova