Jaroslava Fiedlerova
f12b8cb44d
CI: rework log analysis to simplify and allow new use cases
The CI has some log analysis, but it is unmaintainable (huge functions
that mix eNB/gNB or lteUE/nrUE, are not tested, most of it is not used
or useful...) or limited to specific use cases (the timing-phytest time
analysis is hardcoded to source builds instead of also handling analysis
when running from docker images). This MR reimplements both functions.
For the log analysis, the CI team decided to throw away the old
analysis, and reimplement a mechanism that checks the couple of things
we care for.
- Allow to analyze phy-test timing analysis from docker files, and add a
unit test
- Rework existing log analysis to remove useless checks, and make it
more composable
* The undeployment steps (Undeploy_Object(), Terminate_eNB()) take an
<analysis> XML object which then either has a <services> subelement
with check that are whitespace delimited (similar to how other xml
task steps have <services>). For checks that contain whitespace,
they can use <service> (singular) that encodes a single check, such
that e.g., arguments could include spaces.
* Each check is of form service=check[=arguments], where service is
the docker services, check is a checking function (implemented as a
class with run(file, options) function inside cls_loganalysis.py,
and options arguments after the second = that is to be read by run()
(basically, options will contain arguments that can be freeform)
* these definitions can occur multiple times, e.g., it could be
serviceA=Check1 and serviceA=Check2=Args and it will execute both
Check1 and Check2
* if a service is declared, it will execute alse the Default check
which currently checks for an assertion
* Implemented (and inspired from the existing CI analysis) are:
- Default check to check for assertions (will be extended, to e.g.,
address/undefined behavior sanitizer analysis)
- RetxCheck: retransmission check as used in the CI
- LastLineContains to check the last line containes a specific
string
- ContainsString to check a specific string appears in logs
- EndsWithBye: check that towards the end (but not necessarily the
last line), there is a Bye. string (a workaround since we don't
always terminate with 0)
- There is a plan to use return codes after stopping in docker, but most
executables do not end with 0 most of the time, such that we needed to
abandon this plan.
- Additional fixes (see commit messages) and delete unused code
This document outlines how to use the OAI CI framework and gives a quickstart.
[[TOC]]
Quickstart
There is a script main.py that
- reads scenario files from XML
- executes the steps (e.g., deployment of CN/gNB/UE, attach, ping, ...)
To simplify, a script run_locally.sh can be used to run a single scenario
locally. First, download images for gNB&UE (we assume you already downloaded
the CN) and tag them "latest", which is expected by run_locally.sh:
docker pull oaisoftwarealliance/oai-gnb:develop
docker tag oaisoftwarealliance/oai-gnb:develop oai-gnb
docker pull oaisoftwarealliance/oai-nr-ue:develop
docker tag oaisoftwarealliance/oai-nr-ue:develop oai-nr-ue
Now, run the scenario:
cd ~/openairinterface5g/ci-scripts/
./run_locally.sh xml_files/container_5g_rfsim_simple.xml
Output should look like
[2025-08-07 18:07:49,631] INFO: ----------------------------------------
[2025-08-07 18:07:49,631] INFO: Creating HTML header
[2025-08-07 18:07:49,631] INFO: ----------------------------------------
[2025-08-07 18:07:49,766] INFO: ----------------------------------------
[2025-08-07 18:07:49,767] INFO: Starting Scenario: xml_files/container_5g_rfsim_simple.xml
[2025-08-07 18:07:49,767] INFO: ----------------------------------------
[2025-08-07 18:07:49,767] INFO: placing all artifacts for this run in /home/richie/oai/cmake_targets/log/container_5g_rfsim_simple.xml.d/
[...]
[2025-08-07 18:07:49,771] INFO: ----------------------------------------
[2025-08-07 18:07:49,771] INFO: Test ID: 1 (#1)
[2025-08-07 18:07:49,771] INFO: Deploy OAI 5G CoreNetwork
[2025-08-07 18:07:49,771] INFO: ----------------------------------------
it will run an end-to-end test, connecting one UE to the gNB, and ping, before
undeploying. As shown, logs will be under
/home/richie/oai/cmake_targets/log/container_5g_rfsim_simple.xml.d/. The
python code also produces an HTML report in ci-scripts/test_results.html.
Running with local changes
In order to run the tests as above, with local changes, please build images locally.
It is not yet possible to override an existing image with changes made locally to a source file and run it directly through the CI framework. However, it is possible to override an existing image with local changes, and run the scenario manually following the console logs steps provided above. This is described in the 5G RFsimulator README.