CI: rework log analysis to simplify and allow new use cases
The CI has some log analysis, but it is unmaintainable (huge functions
that mix eNB/gNB or lteUE/nrUE, are not tested, most of it is not used
or useful...) or limited to specific use cases (the timing-phytest time
analysis is hardcoded to source builds instead of also handling analysis
when running from docker images). This MR reimplements both functions.
For the log analysis, the CI team decided to throw away the old
analysis, and reimplement a mechanism that checks the couple of things
we care for.
- Allow to analyze phy-test timing analysis from docker files, and add a
unit test
- Rework existing log analysis to remove useless checks, and make it
more composable
* The undeployment steps (Undeploy_Object(), Terminate_eNB()) take an
<analysis> XML object which then either has a <services> subelement
with check that are whitespace delimited (similar to how other xml
task steps have <services>). For checks that contain whitespace,
they can use <service> (singular) that encodes a single check, such
that e.g., arguments could include spaces.
* Each check is of form service=check[=arguments], where service is
the docker services, check is a checking function (implemented as a
class with run(file, options) function inside cls_loganalysis.py,
and options arguments after the second = that is to be read by run()
(basically, options will contain arguments that can be freeform)
* these definitions can occur multiple times, e.g., it could be
serviceA=Check1 and serviceA=Check2=Args and it will execute both
Check1 and Check2
* if a service is declared, it will execute alse the Default check
which currently checks for an assertion
* Implemented (and inspired from the existing CI analysis) are:
- Default check to check for assertions (will be extended, to e.g.,
address/undefined behavior sanitizer analysis)
- RetxCheck: retransmission check as used in the CI
- LastLineContains to check the last line containes a specific
string
- ContainsString to check a specific string appears in logs
- EndsWithBye: check that towards the end (but not necessarily the
last line), there is a Bye. string (a workaround since we don't
always terminate with 0)
- There is a plan to use return codes after stopping in docker, but most
executables do not end with 0 most of the time, such that we needed to
abandon this plan.
- Additional fixes (see commit messages) and delete unused code
GPU Channel Simulation: nr_dlsim/nr_ulsim Integration + CI Test
This MR is a follow-up to !3886 (merged) and extends CUDA-based channel
simulation support to nr_dlsim and nr_ulsim, together with the
corresponding CI tests.
The MR incorporates:
- Cherry-picked commits e0c204f7 and 5f344876 from !3588
- Integration of new CI physim tests using GPU-accelerated channel
simulation in nr_dlsim and nr_ulsim tests, see commit c727b3bb
New CI job: RAN-Channel-Simulation
https://jenkins-oai.eurecom.fr/job/RAN-Channel-Simulation/
[CI] Pull Ubuntu base image from internal registry
This MR updates the Docker build process for OAI images to use a private
registry for the Ubuntu base image ubuntu:noble. This will reduce the
possibility for CI disruption if no internet connection exists.
- Replace FROM ubuntu:noble with FROM <DEFAULT_REGISTRY>/ubuntu:noble in Ubuntu Dockerfiles
This change overcomes the pull rate limit issues from DockerHub in the
CI.
The <DEFAULT_REGISTRY> contains a multi-architecture image (arm64 and
amd64)
$ docker buildx imagetools inspect gracehopper3-oai.sboai.cs.eurecom.fr/ubuntu:noble | grep Platform
Platform: linux/amd64
Platform: linux/arm64
- The scripts run from the jenkins server everyday using a crontab.
- It pulls the AMD and ARM ubuntu:noble image on respective servers and
checks if the internal registry is in sync.
- If not, it pushes the ubuntu:noble image to the registry to remain in
sync with dockerhub.
It was initially foreseen to just check the program exit code, but most
executables don't return with 0 (so the CI would always fail). Add a
separate check that will check for the Bye., only going through the last
X lines instead of the entire file.
- we agreed to basically throw away the old log checker (as it is
unmaintainable)
- now: provide list of analysis to do
* all checkers in cls_log_analysis, can be unit tested (see also next
commit)
* looked up dynamically to simplify adding of new tests + config
* configured with
<analysis>
<services>DESCRIPTION [DESCRIPTION...]</services>
<service>DESCRIPTION</service>
...
</analysis>
where DESCRIPTION follows service=func[=options]:
- service is name of service to check
- func is function to call, which receives filename
- option are arbitary options to pass to func
<services> is whitespace delimited (so can take multiple service
analysis definitions)
<service> exists to allow service definitions with whitespace
changes that would not work in <services>
- I initially planned to check return code, but most softmodems actually
exit with non-zero return code, so this is still TODO
- Default analyzer (checking for assertions, ...) is always run if a
service is listed
- checks for file size
- add unit tests
- it is spaghetti code: difficult to understand which code paths are
taken
- the only effect seems to be in the UE log analysis function: it might
make the log fail. However, we test that iperf is successful
(Iperf2_Unidir()), and would detect before the log analysis if MBMS
traffic does not work
- the rest is only messages that likely nobody reads
The real reason for this change is to reduce the number of class members
in the ran.py class, to simplify the functioning of the eNB analysis
function (no self) to reduce dependencies on the ran.py class.
set -e makes the script exit on error. However, there is code to handle
a non-zero return code of the script, and -e prevents from writing the
footer.
add set -x to see all the commands that are being executed.
Use a raw string as suggested:
ci-scripts/cls_static_code_analysis.py:93: SyntaxWarning: invalid escape sequence '\:'
result = re.search('([a-zA-Z0-9\:\-\.\/])+\.git', self.ranRepository)
ci-scripts/cls_static_code_analysis.py:125: SyntaxWarning: invalid escape sequence '\.'
ret = re.search('cppcheck version="(?P<version>[0-9\.]+)"', str(line))
ci-scripts/cls_oai_html.py:221: SyntaxWarning: invalid escape sequence '\/'
cmd = "sed -i -e 's/__STATE_" + self.htmlTabNames[0] + "__/<span class=\"glyphicon glyphicon-remove\"><\/span>/' test_results.html"
ci-scripts/ran.py:428: SyntaxWarning: "\[" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\["? A raw string is also an option.
result = re.search('\[gNB [0-9]+\]\[RAPROC\] PUSCH with TC_RNTI 0x[0-9a-fA-F]+ received correctly, adding UE MAC Context RNTI 0x[0-9a-fA-F]+', str(line))
ci-scripts/ran.py:457: SyntaxWarning: "\[" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\["? A raw string is also an option.
result = re.search('\[PHY\]\s+problem receiving samples', str(line))
ci-scripts/ran.py:461: SyntaxWarning: "\[" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\["? A raw string is also an option.
result = re.search('\[MAC\]\s+Removing UE', str(line))
I am not sure we ever need the millisecond output. The date is typically
also clear from the output (we could print the date at the beginning,
once, instead, but this is not done here. Finally, in Jenkins, we
typically get time stamps. Thus, it's not necessary to print this by
default.
On the other hand, since it might be useful when running locally, make
time stamps in logs configurable.
Fix compilation error:
/oai-ran/openair1/SIMULATION/NR_PHY/dlsim.c:932:42: error: variable 'h_channel_coeffs' set but not used [-Werror,-Wunused-but-set-variable]
932 | float *h_channel_coeffs = NULL;
| ^
- Introduce usage of the -f option in ulsim to allow enabling specific
feature flags at runtime.
- Add support for the "cuda" flag to enable CUDA channel simulation when
compiled with CUDA support. If an unsupported flag is provided, nr_ulsim
reports an error.
This commit integrates the GPU-based channel simulation pipeline into
the ulsim.
A `--cuda` command-line flag is added to enable the GPU path at
runtime. When activated, the simulator will:
- Allocate all necessary GPU and pinned host memory via the
`init_cuda_chsim_buffers` helper function.
- In the main simulation loop, call the `run_channel_pipeline_cuda`
function to perform the channel simulation on the GPU, bypassing the
CPU-based functions.
- Clean up all CUDA resources on exit using the
`free_cuda_chsim_buffers` helper.
The cpu path is also updated to use multipath_channel_float and add_noise_float.
- Replace previous handling of -f with a generic feature flag option.
- Add support for the "cuda" flag to enable CUDA channel simulation when
compiled with ENABLE_CUDA.
- Remove unused SCGroupConfig file handling (scg_fd) and related code.
- Clean up deprecated code and variables associated with SCG reading - not
required anymore after b232be94ed
Fixes for Initial UE Context Setup
This MR fixes DRB aggregation from E1 so DRBs from multiple PDU sessions
are appended correctly into the F1 DRB array instead of overwriting
existing entries. It also makes NGAP Initial Context Setup skip PDU
sessions that are already established for the UE, only setting up new
sessions.
DRB aggregation from E1: Adjust fill_drb_to_be_setup_from_e1_resp to
pass &drbs[nb_drb] into fill_f1_drbs_from_e1, ensuring DRBs from later
PDU sessions are appended after the current count.
Duplicate PDU sessions in NGAP Initial Context Setup: In
rrc_gNB_process_NGAP_INITIAL_CONTEXT_SETUP_REQ, detect requested PDU
session IDs that already exist with PDU_SESSION_STATUS_ESTABLISHED, log
a warning, and skip them so they are not added to UE->initial_pdus.
Synchronize multiple gNBs by aligning the slot boundaries
Currently, we derive frame and slot number based on proc->timestamp_rx
using
proc->frame_rx = (proc->timestamp_rx / (fp->samples_per_subframe*10))&1023;
proc->tti_rx = get_slot_from_timestamp(proc->timestamp_rx, fp);
This ignores the samples
proc->timestamp_rx % fp->samples_per_frame
while computing the frame and slot number creating a mis-alignment with
the slot boundaries. A detailed description of the slot misalignment
in a pictorial representation can be seen in an attachment in the MR [1].
We fix this alignment issue by compensating these samples during the
first reception.
We can test the synchronization of the multiple gNBs from the estimated
ToAs from the PRS in the Downlink and the estimated ToAs from the SRS in
the Uplink using rfsim.
In order to test this, make sure you build gNB and nrUE along with the
telnetsrv flag using the option --build-lib telnetsrv.
Also, enable the following log in the function
nr_est_srs_timing_advance_offset() in
openair1/PHY/NR_ESTIMATION/nr_measurements_gNB.c to observe SRS ToAs
LOG_I(NR_PHY, "SRS estimatd ToA %d [RX ant %d]: TA offset %d, TA offset ns %d (max_val %ld, mean_val %ld, max_idx %d)\n", srs_toa, ant, *ta_offset, *ta_offset_nsec, max_val, mean_val, max_idx);
Furthermore, update the option Active_gNBs = 2; in
targets/PROJECTS/GENERIC-NR-5GC/CONF/ue.nr.prs.fr1.106prb.conf to enable
decoding PRS transmitted from two gNBs
To run:
nrUE:
sudo ./nr-uesoftmodem -r 106 --numerology 1 --band 78 -C 3619200000 --uicc0.imsi 001010000000001 --rfsim --phy-test -O ../../../targets/PROJECTS/GENERIC-NR-5GC/CONF/ue.nr.prs.fr1.106prb.conf --rfsimulator.serveraddr server
gNB 0:
sudo ./nr-softmodem -O ../../../targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb0.prs.band78.fr1.106PRB.usrpx310.conf --gNBs.[0].min_rxtxtime 6 --rfsim --phy-test --rfsimulator.[0].serveraddr 127.0.0.1 --rfsimulator.[0].options chanmod --rfsimulator.[0].modelname AWGN --telnetsrv
gNB 1:
sudo ./nr-softmodem -O ../../../targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb1.prs.band78.fr1.106PRB.usrpx310.conf --gNBs.[0].min_rxtxtime 6 --rfsim --phy-test --rfsimulator.[0].serveraddr 127.0.0.1 --rfsimulator.[0].options chanmod --rfsimulator.[0].modelname AWGN --telnetsrv
When we run the nrUE, gNB0 and gNB1, we can see the following logs
nrUE:
[PHY] [gNB 0][rsc 0][Rx 0][sfn 976][slot 2] DL PRS ToA ==> 0.0 / 2048 samples, peak channel power -16.4 dBm, SNR +4.0 dB, rsrp -42.2 dBm
[PHY] [gNB 1][rsc 0][Rx 0][sfn 976][slot 3] DL PRS ToA ==> 0.0 / 2048 samples, peak channel power -16.4 dBm, SNR +4.0 dB, rsrp -42.2 dBm
gNB 0:
[NR_PHY] SRS estimatd ToA 0 [RX ant 0]: TA offset 31, TA offset ns 0 (max_val 99820162, mean_val 80176, max_idx 0)
gNB 1:
[NR_PHY] SRS estimatd ToA 0 [RX ant 0]: TA offset 31, TA offset ns 0 (max_val 98784562, mean_val 79369, max_idx 0)
From the above logs, when we run the rfsim and when no delays introduced, we can observe the following results
In the Downlink at the UE, the estimated ToAs using PRS from both gNB0
and gNB1 are 0 and in the uplink, the ToA estimated from the SRS
transmitted by the UE at both the gNBs are also 0. This indicates that
both the gNBs are synchronized along with their slot boundaries. We also
highlight that this is not the case in the current develop branch since
the slot boundaries are not aligned.
We also can test the multi gNB synchronization by introducing delays
using telnet as follows,
Introduce new_offset (delay) of 4 samples to the gNB 0 from UE in the
uplink as follows,
openairinterface5g$ echo rfsimu setdistance rfsimu_channel_enB0 20 | ncat --idle 1 127.0.0.80 9080
softmodem_gnb> rfsimu_setdistance_cmd: new_offset 4, new (exact) distance 19.518 m, new delay 0.000065 ms
Ncat: Idle timeout expired (1000 ms).
We can oberve this delay of 4 samples being reflected in the estimated
SRS ToAs at gNB 0 while not changing the ToA in gNB 1
[NR_PHY] SRS estimatd ToA 4 [RX ant 0]: TA offset 31, TA offset ns 65 (max_val 98803636, mean_val 79430, max_idx 8)
similarly, Introduce new_offset (delay) of 2 samples to the gNB 1 from
UE in the uplink as follows,
openairinterface5g$ echo rfsimu setdistance rfsimu_channel_enB0 10 | ncat --idle 1 127.0.0.81 9081
softmodem_gnb> rfsimu_setdistance_cmd: new_offset 2, new (exact) distance 9.759 m, new delay 0.000033 ms
Ncat: Idle timeout expired (1000 ms).
We can oberve this delay of 2 samples being reflected in the estimated
SRS ToAs at gNB 1 while not changing the ToA in gNB 0
[NR_PHY] SRS estimatd ToA 2 [RX ant 0]: TA offset 31, TA offset ns 32 (max_val 98962768, mean_val 79552, max_idx 4)
Closes#897
[1] https://gitlab.eurecom.fr/oai/openairinterface5g/-/merge_requests/3892
Remove L1 dependence from NR band number
L1 band information relied on get_band function that couldn't provide an
unambiguous result. Also there is no SCF compliant parameter for L1 and
generally we don't really need that information in L1.
The MR also remove a RU section that used nr_band but was never really
accessed in NR.
Add missing packing for CONFIG.request parameter BetaPSS
Adds the packing procedure for the parameter BetaPSS, as well as adding this
parameter to the utility functions where it is missing
IPV6 support for telnet socket binding
Now, the listening telnet socket is only bound to the dedicated interface IPs in
config file, both for V4 and V6
Closes#834
Fix handling of IPv4v6 PDU sessions
The code previously checked that the supplied PDU sessions's type is
exactly what we requested. However, at least OAI 5GC will only give us
an IPv4 PDU session if we request IPv4v6 (so the check would fail).
Extend the validation to either check exact match, or, if IPv4v6
requested, that it is either IPv4 or IPv6.
Jaroslava Fiedlerova