- pass parameter directly instead of indirectly via class
- harmonize to svr_id to read through common code in the next commits
- pass images to express which images will be deleted
- refactor function to use "with" for cls_cmd, and return success (or
not)
- pass parameter directly instead of indirectly via class
- call it simply "images", to harmonize with clean-function later
- harmonize to svr_id as in the previous commit
- rewrite function to use with-statement with cls_cmd
- optionally pass a tag to use (which will be used in a test, and can be
used to pull flexric)
- Introduce Pull_Image() which does the actual pulling, can pull from
any registry, and will be reused in the case of pulling from cluster.
- pass parameter directly instead of indirectly via class
- harmonize to svr_id to harmonize parameter with parameters in the next
commit (and towards a future harmonization of directly taking node
names instead of "IDs")
In the last commits, SSHconnection class has been removed from
cls_containerize.py. We therefore don't need the username and password,
as cls_cmd uses certificates only. Remove the usage of the username and
the password, and harmonize the use of GetCredentials().
Recent refactoring replaced the CreateWorkspace function call with a
dedicated XML step (for clarity, and because it avoids needless
recreation of workspaces). In the proxy function, the same
CreateWorkspace xml step has been added, cloning the OAI RAN repo, when
we actually need the repository for the proxy. Hence, fix this by
calling the CreateWorkspace() function manually, but only when
necessary, to build the proxy. (It would only be a problem if the
pipeline would rebuild the proxy, which did not happen because not much
work is happening on the proxy repo).
Further, refactoring and simplication of the proxy build function were
done:
- never tag proxy with develop, that makes no sense
- overwrite CreateWorkspace() info after having created and used the
workspace (the "if" to build the proxy)
- simplify copying of logs
- add simple unit test
- remove check for ubuntu; the dependcy is "docker". Simplify the code
to not obscure with a class variable.
Remove CheckAndAddRoute(), as it is complicated to read, uses a
multitude of hostname/IP/password combinations. Replace with scripts on
the machines, which, as of this commit, are:
- porcepix: ~/after-reboot.sh
- ofqot: ~/after-reboot.sh
- nepes: ~/after-reboot.sh
- obelix: ~/how-to-resetup-obelix-out-of-reboot.sh
Since commit 041b2b2d7a, the CI main loop
iterates over all steps, even if there has been a problem during an
earlier XML step. In all cases, UndeployObject() is marked to always
run, as we always need to make sure that all deployed services are
terminated. UndeployObject() always recovers logs. Hence, it's not
necessary to do it here as well; it leads to duplicated logs.
This is visible in all pipeline with a failed deployment: the logs
appear twice.
chore(ci): creating a develop tag on top of develop-commit-id tag for the local private registry
Create a 2nd tag of develop for all images on the local registry. We
are doing this in order to simplify cross-repository testing between
FlexRIC and RAN.
Fix Regression 4G
- Fix various bugs and/or regressions in 4G code
- Also, the CI does not test RFsim 4G with --sanitize, enables this to
detect problems.
pyflakes warns about various errors, addressed here. For cls_cluster.py,
remove (unknown) sys.exit and replace through Exception, as sys.exit()
has the unwanted effect that the CI script would stop. Instead, raise
an Exception that would show the problem in the HTML, and ensures that
the script runs until the end.
These are the warnings flagged by pyflakes.
cls_cluster.py:297:4: undefined name 'sys'
cls_containerize.py:39:1: 'subprocess' imported but unused
cls_containerize.py:41:1: 'threading' imported but unused
cls_containerize.py:43:1: 'multiprocessing.Process' imported but unused
cls_containerize.py:43:1: 'multiprocessing.Lock' imported but unused
cls_containerize.py:43:1: 'multiprocessing.SimpleQueue' imported but unused
cls_containerize.py:49:1: 'cls_cluster as OC' imported but unused
cls_containerize.py:50:1: redefinition of unused 'cls_cmd' from line 42
cls_module.py:28:1: 'os' imported but unused
cls_module.py:29:1: 'sys' imported but unused
cls_module.py:34:1: 'subprocess' imported but unused
cls_module.py:35:1: 'datetime.datetime' imported but unused
cls_module.py:211:11: undefined name 'sshconnection'
cls_oaicitest.py:37:1: 'pexpect' imported but unused
cls_oaicitest.py:47:1: 'cls_cluster as OC' imported but unused
cls_oaicitest.py:54:1: 'matplotlib.pyplot as plt' imported but unused
cls_oaicitest.py:55:1: 'numpy as np' imported but unused
cls_oai_html.py:34:1: 'sys' imported but unused
cls_oai_html.py:41:1: 'multiprocessing.Process' imported but unused
cls_oai_html.py:41:1: 'multiprocessing.Lock' imported but unused
cls_oai_html.py:41:1: 'multiprocessing.SimpleQueue' imported but unused
cls_physim.py:37:1: 'multiprocessing.SimpleQueue' imported but unused
cls_static_code_analysis.py:39:1: 'time' imported but unused
cls_static_code_analysis.py:40:1: 'multiprocessing.Process' imported but unused
cls_static_code_analysis.py:40:1: 'multiprocessing.Lock' imported but unused
cls_static_code_analysis.py:40:1: 'multiprocessing.SimpleQueue' imported but unused
epc.py:41:1: 'multiprocessing.Process' imported but unused
epc.py:41:1: 'multiprocessing.Lock' imported but unused
epc.py:41:1: 'multiprocessing.SimpleQueue' imported but unused
main.py:60:1: 'pexpect' imported but unused
main.py:66:1: 'datetime' imported but unused
main.py:68:1: redefinition of unused 'subprocess' from line 63
main.py:70:1: 'multiprocessing.Process' imported but unused
main.py:70:1: 'multiprocessing.Lock' imported but unused
main.py:70:1: 'multiprocessing.SimpleQueue' imported but unused
provideUniqueImageTag.py:2:1: 'os' imported but unused
Use a script, which can be tested independently of CI code (by running
it on the host on which it will be used), and print the directory that
has been prepared in the HTML. Also, in the description, use the
infinitive/imperative(?) of the verb to harmonize with other steps.
This reimplements the CI deployment functions, and makes them a bit more
robust. Concretely:
- Introduce a generic "deployment tag" option that can be used to
override the tag to use for specific images. By default, it is the
current "branch-commitID[0:8]" tag name (requires the change from
function ImageTagToUse() to CreateTag(), impacting also pull/push
image functions)
- Avoid sed for image names, and use an .env file that docker-compose
picks up automatically; the deployment analyzes a potentially existing
.env file and updates instead of simply overriding. For instance, some
pipelines might use -asan images for the gNB and "normal" (non-asan)
images for UE, and a simple overwriting might make the -asan image
name tag disappear for the gNB, resulting in deployment failures).
Finally, undeployment removes the .env file, so that there are no
modifications in the repository after undeployment.
- Redo the previous behavior of autodetecting asan, and use that (the
current function always uses asan, no matter what)
- Remove deployKind/displayedNewTags globals, as they are not necessary
- Make the usedImage output in HTML slimmer
- On undeployment, print service names next to undeploy analysis, and
return success/failure.
- Make the functions generally less verbose and easier to read
Note that as of and only in this commit, deployment does not work, as
all the YAML files have not been updated to work with this updated
version. The next commit adds tests for the new deployment, and updates
one YAML file (also being used in the tests). The follow-up commit then
modifies all YAML files.
Use the previous commit on returning success from each task execution
function, and implement handling of errors in the main loop. Add a
script with which this is visible using Custom_Commands.
Create_Workspace is used used in XML files instead of directly being
executed in DeployObject(). This can be problematic if we deploy
multiple services on the same host, since a previous workspace will be
deleted. Also, it removes futile git clones.
There are other XML steps (CppCheckAnalysis, LicenceAndFormattingCheck,
BuildImage, BuildProxy, BuildClusterImage) that implicitly called
Create_Workspace. Those calls have been removed and care has been taken
to insert the corresponding Create_Workspace steps in the XML.
Update instructions and CI for cross-compilation on Ubuntu 22.
Note that for the dockerfile, we use egrep -L which has changed and does
not exit with code 0 on no match. Instead, use -vqz to invert return
code (0 if nothing found, otherwise 1).
This updates all images in the CI to Ubuntu 22. It also updates
corresponding documentation. Remove python(2) from the image, as it is
not needed. Update documentation.
It seems that the gcc coming in Ubuntu 22 does not like
stack-protector-strong, so remove it as well.
Since we compile Flexric, which does not support Ubuntu 22's default
gcc-11 (it hits a compile bug), switch to gcc-12 manually.
Since we update the compiler, we also need to be careful to use the right
address sanitizer version (which is bundled with the compiler version).
To avoid a reinstallation of the wrong asan version in target images, we
modify Ubuntu 22 targets to copy libasan from the base image. This is necessary
since we use update-alternative in the base image, but we don't want to
install compilers in the target images to use update-alternatives to
install the right asan version... This should be undone when switching
to Ubuntu 24 in the future, as it will have a newer default gcc version,
such that we don't need to update the compile in the first place.
Upgrade libpython in lteUE for U22 upgrade to make it compatible with
UHD.
Robert Schmidt