In CI we sometimes encouter fail of the gNB/UE deployment (and fail of
the pipeline) caused by unsuccsessful initialization of the USRP N310.
Restart the container if the health check fails during initialization.
Introduce a configurable parameter (from XML) to set max number of
attempts for the container deployment.
Store logs from failed deployment attempts.
Rework GetContainerHealth() to work with docker compose and services,
instead of using docker and containers. Return health and message about
deployment status (used for console and HTML logging) for a given service.
chore(ci): adding Flexric-RAN integration test
- Triggering a standalone FLEXRIC + 5G-RAN pipeline: OAI-FLEXRIC-RAN-Integration-Test
- will be triggered by the RAN-Container-Parent
- but also by the OAI-FLEXRIC pipeline (to be done once integrated)
- Added an new python option: --FlexRicTag= than if not used, won't have any effect
- Nice addition:
- if --FlexRicTag=develop or --ranCommitID=develop --> will use the latest commit
on either dev or develop branch.
- Scenario-wise
- Using the cn5g deployment file from Luis in documentation, at least we will use
it to make sure it is working at all time
- We can add more test and especially checkers on the flexric calls
Linked to !3085 (merged)
Occasionally, the CN deployment takes longer than 60 seconds to complete
successfully. As a result, the CI reports failure, even when all other test
stages pass (e.g., gNB attaches to the CN, UE connects, and iperf tests succeed).
Replace current docker deployment of OAI CN by deployment on OC. With change
of the CN, we also need to change the routing for UE module - change the
mbim script for controling the UE.
Reduce double-usage of CI XML files
Some pipelines use the same XML files. For instance,
RAN-NSA-B200-Module-LTEBOX-Container and RAN-LTE-{TDD,FDD}-LTEBOX-Container
use both fr1_epc_{closure,start}.xml, but use the core network on different
hosts, by means of different pipeline configuration (EPCtype, etc).
A future MR will remove these additional pipeline configuration, and store
all the CN config in the repository. To simplify the transition, duplicate
some XMLs (i.e., remove duplicate usage of them).
After this MR, the only "double usage" of XML files for core (un-)deployment
should be through pipelines RAN-LTE-{TDD,FDD}-LTEBOX-Container.
If we are pulling FlexRIC, overwrite the tag to use with what has been
provided on the command line. This is because we hardcode the tag schema
"branchname-commitId[0:8]" everywhere else, but this does not apply for
FlexRIC, and we have to overwrite this here.
- pass parameter directly instead of indirectly via class
- harmonize to svr_id to read through common code in the next commits
- pass images to express which images will be deleted
- refactor function to use "with" for cls_cmd, and return success (or
not)
- reuse cls_containerize.Containerize.PullImage()
- pass parameter directly instead of indirectly via class
- always read node name and use this instead of "sometimes" looking it
up from a class member. With "sometimes" I mean that if no node name
is given, it will take it from self.eNBIPAddress, which is confusing
at best if we can give the node name directly
- consistently put the node name in the xml file
- pass parameter directly instead of indirectly via class
- call it simply "images", to harmonize with clean-function later
- harmonize to svr_id as in the previous commit
- rewrite function to use with-statement with cls_cmd
- optionally pass a tag to use (which will be used in a test, and can be
used to pull flexric)
- Introduce Pull_Image() which does the actual pulling, can pull from
any registry, and will be reused in the case of pulling from cluster.
The CI pipeline RAN-NSA-B200-Module-LTEBOX-Container uses the same core
network (ltebox) as the pipelines RAN-LTE-{TDD,FDD}-LTEBOX-Container,
triggered through the same XML files. However, the core networks are on
different hosts, configured through (Jenkins) pipeline configuration. A
future MR will move the entire core network configuration into the XML
files [1]. To facilitate this, create separate XML files for the NSA
pipeline.
[1] The advantage is that it will be easier to reason with only the XML
files what is going to be run where and how.
- pass parameter directly instead of indirectly via class
- harmonize to svr_id to harmonize parameter with parameters in the next
commit (and towards a future harmonization of directly taking node
names instead of "IDs")
There seem to be randomly occurring deadlocks when running the UE in the
CI. MR !3091 should fix this. While waiting for this, apply a workaround
by increasing the number of worker pools in the UE to 12.
/opt/oai-gnb/bin/nr-softmodem -O /opt/oai-gnb/etc/gnb.conf --sa --log_config.global_log_options level,nocolor,time --rfsimulator.options chanmod --telnetsrv --telnetsrv.listenaddr 192.168.71.150 --telnetsrv.shrmod ci
=================================================================
==7==ERROR: AddressSanitizer: odr-violation (0x7fce78b85be0):
[1] size=4 'log_mem_write_side' /oai-ran/common/utils/LOG/log.c:78:14
[2] size=4 'log_mem_write_side' /oai-ran/common/utils/LOG/log.c:78:14
These globals were registered at these points:
[1]:
\#0 0x7fce7b691658 in __asan_register_globals ../../../../src/libsanitizer/asan/asan_globals.cpp:341
\#1 0x7fce7bd0c47d in call_init elf/dl-init.c:70
[2]:
\#0 0x7fce7b691658 in __asan_register_globals ../../../../src/libsanitizer/asan/asan_globals.cpp:341
\#1 0x7fce7a5e2eba in call_init ../csu/libc-start.c:145
\#2 0x7fce7a5e2eba in __libc_start_main_impl ../csu/libc-start.c:379
==7==HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_odr_violation=0
SUMMARY: AddressSanitizer: odr-violation: global 'log_mem_write_side' at /oai-ran/common/utils/LOG/log.c:78:14
==7==ABORTING
[INFO tini (1)] Spawned child process '/opt/oai-gnb/bin/entrypoint.sh' with pid '7'
[INFO tini (1)] Main child exited normally (with status '1')
* trigger re-establshemnt on DU: reset RLC counters on DRB 1
for OAI UE, since SRB 1 does not work for lack of periodic
measurements reports
* how is the re-establishment triggered?
* a transmission on DRB 1 occurs
* detection of a mismatch of RLC counters at the UE side
* max number of RLC retransmissions reached
* RLF detection
* verify re-establishment on CU: we ping the core from the UE; this
sends packets on DRB, and because the sequence number mismatches, will
trigger a reestablishment. We accept in the ping that a couple of
packets might be lost; after ping is over, reestablishment must be
done as well (else the ping will already fail).
CI: Cleanup of cls_physim.py
- Remove unused code and file
- Rework functions for CUDA and T2 log analysis
- Check if test was successful for nr_ulsim and nr_dlsim in
RAN-gNB-N300-Timing-Phytest-LDPC pipeline
Set SA mode as default
- Set standalone (SA) mode as the default.
- Adjust commands in Docker Compose files by removing --sa flags and
adding --nsa flags where applicable.
- Add a function to verify that only one mode is being used at a time.
- Update all relevant documentation.
In the last commits, SSHconnection class has been removed from
cls_containerize.py. We therefore don't need the username and password,
as cls_cmd uses certificates only. Remove the usage of the username and
the password, and harmonize the use of GetCredentials().
Recent refactoring replaced the CreateWorkspace function call with a
dedicated XML step (for clarity, and because it avoids needless
recreation of workspaces). In the proxy function, the same
CreateWorkspace xml step has been added, cloning the OAI RAN repo, when
we actually need the repository for the proxy. Hence, fix this by
calling the CreateWorkspace() function manually, but only when
necessary, to build the proxy. (It would only be a problem if the
pipeline would rebuild the proxy, which did not happen because not much
work is happening on the proxy repo).
Further, refactoring and simplication of the proxy build function were
done:
- never tag proxy with develop, that makes no sense
- overwrite CreateWorkspace() info after having created and used the
workspace (the "if" to build the proxy)
- simplify copying of logs
- add simple unit test
- remove check for ubuntu; the dependcy is "docker". Simplify the code
to not obscure with a class variable.
The first thing Build_Proxy does is to remove the repository. Further,
Build_Proxy does not use the OAI RAN repo, but the proxy repo. The
Create_Workspace() is therefore completely useless.
Remove CheckAndAddRoute(), as it is complicated to read, uses a
multitude of hostname/IP/password combinations. Replace with scripts on
the machines, which, as of this commit, are:
- porcepix: ~/after-reboot.sh
- ofqot: ~/after-reboot.sh
- nepes: ~/after-reboot.sh
- obelix: ~/how-to-resetup-obelix-out-of-reboot.sh
It is clearer in CI logs to use absolute paths, and that is also a
"preference" of the new SSH class (cls_cmd.py) in the sense that some
functions don't handle "cd" gracefully (because of underlying Python
library paramiko).
For zip, though, we should change to have nice relative paths in the
archives, so use cd <path> && zip ... to achieve that, instead of
changing to a directory. This way, the path on which we operate is still
visible in the logs in each line.
- Introduce "Run_Physim" function for running ldpctest (with CUDA) and
nr_dlsim/nr_ulsim (with T2) in RAN-gNB-N300-Timing-Phytest-LDPC test
pipeline.
- Create "cls_analysis.py" and introduce "analyze_physim" for
log analysis of physim tests.
- Define patterns to be searched in the physim logs.
- Extract and print information about total gNB RX/TX PHY processing time
from nr_ulsim/nr_dlsim logs.
- Remove "cls_physim.py" as it is not used anymore, cleanup "main.py".
Jaroslava Fiedlerova