After the parent commit, the registry might contain prefixed images.
This commit adds the functionality to pull such prefixed image. Since we
rename the pulled image (to have a consistent name, from whichever
registry we might pull), remove the prefix as well so that it can be
used with other CI functionality (remove image, use in test, ...)
Examples for renaming:
- pull from internal_registry for x86
internal_registry/oai-gnb:branchA => oai-ci/oai-gnb:branchA
- pull from internal registry for ARM with "arm_" prefix
internal_registry/oai-gnb:arm_branchA => oai-ci/oai-gnb:branchA
- pull from x86 other registry (e.g., openshift)
openshift/namespace/oai-gnb:branchA => oai-ci/oai-gnb:branchA
The only currently viable way to push ARM images to our registry is to
use a custom tag prefix. Modify the python test code to specify such a
tag, and default to "" (no prefix).
After the preceding commits, it's now possible to build images on ARM.
Add a specific "native_arm" build kind to only build what we need as of
now (do not only call it "arm", as that would match the kind
"build_cross_arm64").
Add the corresponding XML. Pushing of images is disabled, as this does
not work as of now.
The (existing) Aerial pipeline on devkit uses an older version of nvipc,
which is upgraded on gracehopper. Use a glob to match them equally,
which also aligns this code with the corresponding Dockerfile, which
already uses that glob.
Local_Cmd forces use to use absolute paths for copy operations (because
that makes more clear where files are being copied). Use the absolute
path to avoid the corresponding exception.
In CI we sometimes encouter fail of the gNB/UE deployment (and fail of
the pipeline) caused by unsuccsessful initialization of the USRP N310.
Restart the container if the health check fails during initialization.
Introduce a configurable parameter (from XML) to set max number of
attempts for the container deployment.
Store logs from failed deployment attempts.
Rework GetContainerHealth() to work with docker compose and services,
instead of using docker and containers. Return health and message about
deployment status (used for console and HTML logging) for a given service.
- pass parameter directly instead of indirectly via class
- harmonize to svr_id to read through common code in the next commits
- pass images to express which images will be deleted
- refactor function to use "with" for cls_cmd, and return success (or
not)
- pass parameter directly instead of indirectly via class
- call it simply "images", to harmonize with clean-function later
- harmonize to svr_id as in the previous commit
- rewrite function to use with-statement with cls_cmd
- optionally pass a tag to use (which will be used in a test, and can be
used to pull flexric)
- Introduce Pull_Image() which does the actual pulling, can pull from
any registry, and will be reused in the case of pulling from cluster.
- pass parameter directly instead of indirectly via class
- harmonize to svr_id to harmonize parameter with parameters in the next
commit (and towards a future harmonization of directly taking node
names instead of "IDs")
In the last commits, SSHconnection class has been removed from
cls_containerize.py. We therefore don't need the username and password,
as cls_cmd uses certificates only. Remove the usage of the username and
the password, and harmonize the use of GetCredentials().
Recent refactoring replaced the CreateWorkspace function call with a
dedicated XML step (for clarity, and because it avoids needless
recreation of workspaces). In the proxy function, the same
CreateWorkspace xml step has been added, cloning the OAI RAN repo, when
we actually need the repository for the proxy. Hence, fix this by
calling the CreateWorkspace() function manually, but only when
necessary, to build the proxy. (It would only be a problem if the
pipeline would rebuild the proxy, which did not happen because not much
work is happening on the proxy repo).
Further, refactoring and simplication of the proxy build function were
done:
- never tag proxy with develop, that makes no sense
- overwrite CreateWorkspace() info after having created and used the
workspace (the "if" to build the proxy)
- simplify copying of logs
- add simple unit test
- remove check for ubuntu; the dependcy is "docker". Simplify the code
to not obscure with a class variable.
Remove CheckAndAddRoute(), as it is complicated to read, uses a
multitude of hostname/IP/password combinations. Replace with scripts on
the machines, which, as of this commit, are:
- porcepix: ~/after-reboot.sh
- ofqot: ~/after-reboot.sh
- nepes: ~/after-reboot.sh
- obelix: ~/how-to-resetup-obelix-out-of-reboot.sh
Since commit 041b2b2d7a, the CI main loop
iterates over all steps, even if there has been a problem during an
earlier XML step. In all cases, UndeployObject() is marked to always
run, as we always need to make sure that all deployed services are
terminated. UndeployObject() always recovers logs. Hence, it's not
necessary to do it here as well; it leads to duplicated logs.
This is visible in all pipeline with a failed deployment: the logs
appear twice.
chore(ci): creating a develop tag on top of develop-commit-id tag for the local private registry
Create a 2nd tag of develop for all images on the local registry. We
are doing this in order to simplify cross-repository testing between
FlexRIC and RAN.
Fix Regression 4G
- Fix various bugs and/or regressions in 4G code
- Also, the CI does not test RFsim 4G with --sanitize, enables this to
detect problems.
pyflakes warns about various errors, addressed here. For cls_cluster.py,
remove (unknown) sys.exit and replace through Exception, as sys.exit()
has the unwanted effect that the CI script would stop. Instead, raise
an Exception that would show the problem in the HTML, and ensures that
the script runs until the end.
These are the warnings flagged by pyflakes.
cls_cluster.py:297:4: undefined name 'sys'
cls_containerize.py:39:1: 'subprocess' imported but unused
cls_containerize.py:41:1: 'threading' imported but unused
cls_containerize.py:43:1: 'multiprocessing.Process' imported but unused
cls_containerize.py:43:1: 'multiprocessing.Lock' imported but unused
cls_containerize.py:43:1: 'multiprocessing.SimpleQueue' imported but unused
cls_containerize.py:49:1: 'cls_cluster as OC' imported but unused
cls_containerize.py:50:1: redefinition of unused 'cls_cmd' from line 42
cls_module.py:28:1: 'os' imported but unused
cls_module.py:29:1: 'sys' imported but unused
cls_module.py:34:1: 'subprocess' imported but unused
cls_module.py:35:1: 'datetime.datetime' imported but unused
cls_module.py:211:11: undefined name 'sshconnection'
cls_oaicitest.py:37:1: 'pexpect' imported but unused
cls_oaicitest.py:47:1: 'cls_cluster as OC' imported but unused
cls_oaicitest.py:54:1: 'matplotlib.pyplot as plt' imported but unused
cls_oaicitest.py:55:1: 'numpy as np' imported but unused
cls_oai_html.py:34:1: 'sys' imported but unused
cls_oai_html.py:41:1: 'multiprocessing.Process' imported but unused
cls_oai_html.py:41:1: 'multiprocessing.Lock' imported but unused
cls_oai_html.py:41:1: 'multiprocessing.SimpleQueue' imported but unused
cls_physim.py:37:1: 'multiprocessing.SimpleQueue' imported but unused
cls_static_code_analysis.py:39:1: 'time' imported but unused
cls_static_code_analysis.py:40:1: 'multiprocessing.Process' imported but unused
cls_static_code_analysis.py:40:1: 'multiprocessing.Lock' imported but unused
cls_static_code_analysis.py:40:1: 'multiprocessing.SimpleQueue' imported but unused
epc.py:41:1: 'multiprocessing.Process' imported but unused
epc.py:41:1: 'multiprocessing.Lock' imported but unused
epc.py:41:1: 'multiprocessing.SimpleQueue' imported but unused
main.py:60:1: 'pexpect' imported but unused
main.py:66:1: 'datetime' imported but unused
main.py:68:1: redefinition of unused 'subprocess' from line 63
main.py:70:1: 'multiprocessing.Process' imported but unused
main.py:70:1: 'multiprocessing.Lock' imported but unused
main.py:70:1: 'multiprocessing.SimpleQueue' imported but unused
provideUniqueImageTag.py:2:1: 'os' imported but unused
Use a script, which can be tested independently of CI code (by running
it on the host on which it will be used), and print the directory that
has been prepared in the HTML. Also, in the description, use the
infinitive/imperative(?) of the verb to harmonize with other steps.
This reimplements the CI deployment functions, and makes them a bit more
robust. Concretely:
- Introduce a generic "deployment tag" option that can be used to
override the tag to use for specific images. By default, it is the
current "branch-commitID[0:8]" tag name (requires the change from
function ImageTagToUse() to CreateTag(), impacting also pull/push
image functions)
- Avoid sed for image names, and use an .env file that docker-compose
picks up automatically; the deployment analyzes a potentially existing
.env file and updates instead of simply overriding. For instance, some
pipelines might use -asan images for the gNB and "normal" (non-asan)
images for UE, and a simple overwriting might make the -asan image
name tag disappear for the gNB, resulting in deployment failures).
Finally, undeployment removes the .env file, so that there are no
modifications in the repository after undeployment.
- Redo the previous behavior of autodetecting asan, and use that (the
current function always uses asan, no matter what)
- Remove deployKind/displayedNewTags globals, as they are not necessary
- Make the usedImage output in HTML slimmer
- On undeployment, print service names next to undeploy analysis, and
return success/failure.
- Make the functions generally less verbose and easier to read
Note that as of and only in this commit, deployment does not work, as
all the YAML files have not been updated to work with this updated
version. The next commit adds tests for the new deployment, and updates
one YAML file (also being used in the tests). The follow-up commit then
modifies all YAML files.
Use the previous commit on returning success from each task execution
function, and implement handling of errors in the main loop. Add a
script with which this is visible using Custom_Commands.
Romain Beurdouche