The generated HTML from CI builds so far only shows the number of
errors, but not the errors themselves. This is inconvenient, as this
means that developers have to go to the logs.
This commit modifies AnalyzeBuildLogs(), which already iterates the
entire file, to store the lines containing "error:" (as it would be
emitted by a compiler). The callers of AnalyzeBuildLogs() now also print
the analysis outcome in a more concise manner.
Also, change the return parameter of AnalyzeBuildLogs() to a tuple to make
the variable handling easier, and remove the unused warnings list.
1. Rename oai-flexric to nearRT-RIC. FlexRIC is a project combining nearRT-RIC, xApps, and E2 agent emulators.
2. Instead of undeploying all the containers at the same time, and not checking the logs for xApps and nearRT-RIC, do the following:
- Stop containers (docker compose stop) - one or subgroup of running containers in the following order:
a) xapp-kpm-moni
b) xapp-kpm-rc
c) xapp-gtp-mac-rlc-pdcp-moni
d) oai-nr-ue oai-nr-ue2
e) oai-gnb
f) nearRT-RIC
- Undeploy containers (docker compose down), collect and analyze all the logs.
For some test cases, we might want to stop a service earlier. This
functionality is done by StopObject().
Test log collection is still done by UndeployObject(), which will
collect the logs of all services (at that point, stopped or not).
GetRunningServices() was returning all services, running or stopped.
Hence, rename to reflect what this function does, and rewrite slightly
to better capture what is actually happening. Also, don't return the
container ID, as we don't use it.
Call stop on all services, as we will down the deployment right after.
The list of services is still used to retrieve logs.
Remove a stray "return False", which would sometimes prevent us from
seeing all error messages.
Following the change to use:
image: ${REGISTRY-oaisoftwarealliance/}${GNB_IMG:-oai-gnb}:${TAG:-develop}
the REGISTRY variable must now include the trailing slash.
For clarity, rename to native_armv9 for clarity. The check inside Python
checked for native_arm, and if we introduce native_armv8 (for jetson),
it would also match that.
Instead of making the function a class member, make it an instance
member because (1) it is more aligned with other functions, notably,
run(), and (2) especially for RemoteCmd() it can be faster, as
successive exec_script() calls do not require a new SSH connection.
CI: speed up and simplify build, remove unnecessary Python code, remove --eNB* parameter list
This cleans up the CI framework for running end-to-end tests, and
improves the integration with OAI code. Specifically
- cleanup build_oai and helpers to output all text, which is captured
better by the CI
- build some images in the Cluster-Build in parallel to increase speed
- remove all --eNB{,1,2}{IPAddress,UserName,Password} variables. Those
were indexed through "instance" and "server ID" and are replaced with
the actual node name. I consider further cleanup to introduce a
"global" XML file node name so we don't need this in each step and
instead define this once.
- individual cleanup
It is superseded by the CreateWorkspace() function and task step (which
removes and recreates a workspace), and this option was not actually
used in any function except for the build of unit tests, but no XML
step actually specified this option.
After having transitioned to direct node names in the XML, these
parameters are not relevant anymore.
The best way to view this patch is with
git show <SHA> --ignore-all-space
Pass node directly from XML to these functions
- BuildClusterImage()
- BuildImage()
- BuildProxy()
- BuildRunTests()
instead of reading the "server ID" and looking up from
eNB_IPAddress/eNB1_IPAddress/eNB2_IPAddress.
Improve the proxy build unit test by properly passing ctx information.
Libboost went from 1.74.0 to 1.83.0 alongside Ubuntu update.
GCC version is set as default to be more practical.
In Ubuntu22, apt sources were in .list files, in Ubuntu24 they are in .sources files.
Add --break-system-packages for pip installation to use same logic as before, see the pip update 23.0.1 (2023-02-17).
With Ubuntu24 liboai_device requires libpython3.12
I have to make both together, since they both depend on
AnalyzeBuildLogs().
- remove mkdir for legacy sub-logfiles in build_oai, that does not exist
anymore; also, remove corresponding log analysis, I think we don't use
it and it is overkill (either build works or not)
- manually try to copy "all.txt"
- explicitly list log files to copy, then analyze
- AnalyzeBuildLogs(): pass log files to inspect explicitly
Remove this option: some tests produce an excessive amount of output on
failure (e.g., googletest/benchmark). This seems to overwhelm the Python
CI code. Suppress this output, because we can always manually run this
test. A high-level "this test is ok/nok" should be fine for the CI.
Speed up docker deployment in CI
- Using docker compose up --wait instead of a custom healthcheck script:
from: https://docs.docker.com/reference/cli/docker/compose/up/
--wait Wait for services to be running|healthy. Implies detached mode.
--wait-timeout Maximum duration in seconds to wait for the project to be running|healthy
- Speeding up the healthchecks by reducing the interval from 1-10s to
0.5s during start period
start_period creates a special period after the start of the container
where the healthcheck is executed every start_interval . During this
time the failures do not count towards the retries counter, but
successes change the status to healthy. This allows container to reach
healthy status much sooner.
- Using init: true to speed up docker compose down/stop
`init: true` initializes the docker container with tini or
alternative: tini is used to collect all the processes and correctly
pass the termination signal to them. This allows the docker container
to be stopped without waiting 10 seconds to kill the container
- Smallest example of this is:
services:
test-init:
image: ubuntu:jammy
command: sleep 10d
- This is to avoid the Docker image builder from being confused in case we have multiple nvIPC files in `/opt/nvidia-ipc`
- This occurs when we have an MR to upgrade the cuBB image
After the parent commit, the registry might contain prefixed images.
This commit adds the functionality to pull such prefixed image. Since we
rename the pulled image (to have a consistent name, from whichever
registry we might pull), remove the prefix as well so that it can be
used with other CI functionality (remove image, use in test, ...)
Examples for renaming:
- pull from internal_registry for x86
internal_registry/oai-gnb:branchA => oai-ci/oai-gnb:branchA
- pull from internal registry for ARM with "arm_" prefix
internal_registry/oai-gnb:arm_branchA => oai-ci/oai-gnb:branchA
- pull from x86 other registry (e.g., openshift)
openshift/namespace/oai-gnb:branchA => oai-ci/oai-gnb:branchA
The only currently viable way to push ARM images to our registry is to
use a custom tag prefix. Modify the python test code to specify such a
tag, and default to "" (no prefix).
After the preceding commits, it's now possible to build images on ARM.
Add a specific "native_arm" build kind to only build what we need as of
now (do not only call it "arm", as that would match the kind
"build_cross_arm64").
Add the corresponding XML. Pushing of images is disabled, as this does
not work as of now.
The (existing) Aerial pipeline on devkit uses an older version of nvipc,
which is upgraded on gracehopper. Use a glob to match them equally,
which also aligns this code with the corresponding Dockerfile, which
already uses that glob.
Local_Cmd forces use to use absolute paths for copy operations (because
that makes more clear where files are being copied). Use the absolute
path to avoid the corresponding exception.
Robert Schmidt