CI: speed up and simplify build, remove unnecessary Python code, remove --eNB* parameter list
This cleans up the CI framework for running end-to-end tests, and
improves the integration with OAI code. Specifically
- cleanup build_oai and helpers to output all text, which is captured
better by the CI
- build some images in the Cluster-Build in parallel to increase speed
- remove all --eNB{,1,2}{IPAddress,UserName,Password} variables. Those
were indexed through "instance" and "server ID" and are replaced with
the actual node name. I consider further cleanup to introduce a
"global" XML file node name so we don't need this in each step and
instead define this once.
- individual cleanup
A previous commit changed to use archiveArtifact() to recover CI logs,
and then attempted to "guess" the L1/MAC stats file name through a
regex. However, the format "enb-[0-9]+" is not correct, as
archiveArtifact() puts the task ID ("[0-9]+") BEFORE the file name
(here, "enb"). So, make it more generic and match the final "-enb.log"
("$" means "end of string").
Fixes: 244ea8fc22 ("Use archiveArtifact() for TerminateeNB()")
Many of these tests exit with an exit code != 0. Thus, the CI
automatically logs the result. Rather, write everything including stderr
to the file for analysis.
Also, rewrite the file with ">" instead of appending through ">>". While
appending works because the CI removes files after archiving, it seems
"cleaner" to remove files.
It is superseded by the CreateWorkspace() function and task step (which
removes and recreates a workspace), and this option was not actually
used in any function except for the build of unit tests, but no XML
step actually specified this option.
After having transitioned to direct node names in the XML, these
parameters are not relevant anymore.
The best way to view this patch is with
git show <SHA> --ignore-all-space
Remove this class. It is not safe to use (no return codes, can get out
of sync), hardcodes the use of passwords, and it's API is cumbersone.
More explanations can also be found in !1867. It is superseded by
cls_cmd.py
Since 28ce0dc444 ("Main: pass in ctx"), the CI always collects logs
into cmake_targets/log/. Hence, LogCollecteNB() is never called. We can
remove it.
Further, change the cmake_targets/log/ to ci-log/, which is shorter, and
append a / on the zip path to make sure that only contents is zipped.
Reimplement these functions by
- using cls_cmd
- retain only the necessary steps from the timing-phytest function
- pass in ctx and node information, instead of server ID
- remove all superfluous variables that don't seem to be used
- make some lists (corresponding indices are "server ID") into normal
variables.
Pass node directly from XML to these functions
- BuildClusterImage()
- BuildImage()
- BuildProxy()
- BuildRunTests()
instead of reading the "server ID" and looking up from
eNB_IPAddress/eNB1_IPAddress/eNB2_IPAddress.
Improve the proxy build unit test by properly passing ctx information.
build_oai primarily builds UHD for OAI. Thus, a number of targets (B100,
USRP1/2, ...) can be omitted, as OAI does not run on them. Further,
disable documentation and man pages, which is typically not necessary
for the averages user, and does not need to be installed for containers.
Similarly to the parent, don't redirect log files to a file. It makes
the CI more complicated, and also raises user awareness of what is being
installed.
Refactor ULSCH scheduler towards more TDD pattern flexibility
Refactors ULSCH scheduler, notably to support "UL-heavy" TDD patterns,
e.g., DSUUU. Also allows to run TDD pattern2 with UL slots, which was a
limitation in the previous code.
Increase the number of aggregration level candidates. Since we have many
UEs with concurrent traffic in that pipeline, we should ensure that we
can allocate many DCIs to not waste resources (e.g., not all RBs used
because could not find DCIs for specific UEs).
Add the possibility to use UL slots in TDD pattern 2. For simplicity, we
require the mixed slot to be of the same symbols in pattern 1 & 2. For
UL slots, simply use the right amount of UL TDAs, depending on which
number of UL slots among pattern 1 & 2 is bigger.
One complication is the TDA for the mixed slot. The current code assumes
that k2 has to be equal or larger than (number of DL slots +1) (because
the spec says min k2=2, but we might want to schedule DSUUU). Then,
there could be for instance the following cases:
- only pattern 1: DSUUU DSUUU ...: the mixed slot has to be reached an
entire period in advance
- pattern 1 & 2: DSUUU DDDDD: normal mixed slot k2, e.g., like for 7DS2U
- pattern 1 & 2: DSUUU DDDSU: the mixed slot has to be reached
N_ul2+N_dl1+1 slots in advance
I can't think of more cases right now. To avoid unscheduled slots, I put
an assertion (which could also trivially be taken out).
A new function get_best_ul_tda() allows to select the "best" TDA out of
a number of TDA. Specifically, get_best_ul_tda() considers in how many
RBs a given TDA might be allocated, and multiplies this number of RBs
with the number of symbols in the TDA to form a "score". The TDA with
the best "score" is selected, considering it maximizes resource usage.
In phytest scheduler, we use this function to verify that the given TDA
can be really allocated in all selected slots.
Schedule SRS longer in advance, so that it would also be enough even
with big k2 for UL TDA.
Remove useless types: we do not use the information on sched_srs
anymore, so remove it.
Check that the VRB map is free before making allocations.
Since the parent commit, the TDA for UL scheduling is dynamically
selected, instead of being hardcoded to specific indices. We can thus
also omit the SRS TDA if no SRS is configured.
Also add SRS-specific TDAs if we schedule further into the future.
Add multiple TDAs with k2 larger than the min_rxtxtime, to potentially
allow to allocate all available slots. For this, nr_rrc_config_ul_tda()
inspects the TDD pattern, and allocates as many TDAs as necessary. It
also sorts the TDA list, with smaller k2 and larger symbol lengths
first.
This in turn is now used by get_num_ul_tda() to use a k2 and the slot
configuration to find a suitable TDA, assuming a list as sorted in
nr_rrc_config_ul_tda().
Note that since get_num_ul_tda() does only inspect the available UL
slots, there might be overlap (e.g., when scheduling SRS), which will be
fixed in a follow-up commit.
Also, TDD pattern 2 is not handled here yet, and a follow-up commit will
fix it.
Add a loop to advance slots as determined by current ("DL") slot, and
advance further into the future (for "UL" slots) if TDAs allow. Ensure
to schedule all TDAs if it's the last DL slot, to guarantee a previously
scheduled TDA will also reoccur (i.e., for guaranteeing that
retransmissions can be made).
Note that the check for number of UE being zero is removed, as this
allows to keep "ul_next" close to the "min_next" slot. This way, we can
use fs_get_max(); otherwise, if we did not always execute this loop,
they might diverge far from each other, making the logic inside the loop
more complicated.
This commit introduces a new type "fsn_t" (frame-slot number) that
encapsulates a single frame-slot combination in a single struct. The
helper functions (in gNB_scheduler_ulsch.c) should be moved to a new
file under common/, but we don't do this here to limit the amount of
code changes.
For scheduling multiple ULSCH ahead, we will have to use different k2.
Hence, we will need to check for k2 in the TDA selection as well. The
function get_ul_tda() is renamed to get_num_ul_tda() to reflect that it
might return multiple TDAs (not done in this commit, but in a follow-up
MR). get_num_ul_tda() also returns a pointer to the first suitable TDA.
In the phy-test scheduler, simplify the selection of the TDA to use: do
not hardcode a specific K2, but rather, use the first one (with any k2).
For the multi-UL slot scheduling functionality in a follow-up commit, it
would be important to know if the scheduler allocated any UEs. If
pf_ul() allocated for some UEs, we might run it again on a different
slot. If on the other hand, no UEs were allocated, there is no point in
running it again.
Introduce support for Cat-B RU using Predefined-Beam BF with Aerial L1
This merge introduces the support for BeamID based Beamforming in OAI L2 using FAPI and leveraging Aerial L1. Support for OAI L1 is not in the scope of this MR.
Main changes:
- Introduced a third state for the analog_beamforming parameter.
0=disabled,1=analog_bf,2=oran_bf
- Correctly populated the FAPI BF PDUs (prg_size, num_prgs,
dig_bf_interfaces)
Make Amarisoft UE sim a RedCap UE
Change Amarisoft UE configuration and gNB of the AW2S pipeline to
be/handle RedCap UEs.
The UE should only connect if the gNB handles RedCap UEs. In the logs,
we should see
[MAC] UE with RNTI 3725 is RedCap
which we check for the first two UEs (one normal UE/absence of this
message, one RedCap UE/presence of this message).
Cleanup documentation and interface of the LDPC segment coding libraries
Fix interface: The LDPC segment decoder interface had variables
'harq_pid', 'ulsch_id' and 'C' that are not useful to the segment
decoder. This MR removes these variables from the interface.
Fix documentation: libconfig log do not display anymore by default so we
may better not show them in the documentation to avoid any
misunderstanding.
Remove ldpc_cl library since the library does not work.
Since the parent commit, UE 1 is normal (no RedCap), UE 2 is RedCap.
Verify that the gNB agrees with that and detects the second UE as RedCap
by analyzing some logs.
The first check is a bit elaborate, but basically is
[ $(docker logs | grep XYZ | wc -l) -eq 0 ]
^ ^ ^ ^ ^
1 2 3 4 5
(1) is the if statement, that compares
(4) the number of occurrences
(3) of "RedCap" in the
(2) docker logs
(5) to being equal to
Note for all UEs which number, and whether it is RedCap (so this need to
be kept in sync with the Amarisoft UE sim configuration file in use!).
Change the parameters to be redcap (release: 17, redcap: "redcap") on
some UEs, and modify the path to the updated configuration.
Specifically, make UE 1 normal (no RedCap), UE 2 RedCap, UEs 3&4 normal,
UEs 5&6 RedCap, etc.
RedCap UEs should now only be able to connect if the gNB is handling Redcap
UEs, which is done in the parent commit.
NR UE NTN: consider acceleration on orbital path to calculate N_UE_TA_drift_variant
The ntn-Config ephemerisInfo contains the SAT position and velocity
vector but not the acceleration vector.
But the acceleration magnitude on the orbital path can be calculated as:
t = 2 * pi * r / v
omega = 2 * pi / t = v / r
acceleration = omega² * r = v² / r
Where t is the orbital period, r is the orbital radius, v is the orbital
velocity, omega is the angular velocity. The acceleration direction is
from the SAT position towards the earth center. To calculate the
acceleration towards the UE, this is projected on the direction from the
SAT to the UE.
This is used to calculate the variable N_UE_TA_drift_variant and improve
the TA accuracy.
Workaround for handling 2 SS0 occasions per slot
Search Space 0 monitoring might depend on SSB. In !3332 this dependence
was removed by moving the configuration of SS0 earlier in a workaround
to avoid a segfault in handover. Now, a new workaround is needed to
handle 2 search space occasions per slot.
Redirecting to a file has the problem that it is difficult in the CI to
recover these files. Furthermore, at least with ninja, when running
build_oai interactively, the build output will only take some lines
(because ninja rewrites lines), so the change should not be that big to
users.
One CI function would print the entire output from build_oai to the
console. Since that output might be long now, do not do this. The file
is archived, anyway.
Sometimes the connection fails because of the reboot. Without may_fail,
it would stop the build, which will then run the wrong code, possibly.
Instead, continue; if there is a problem with USRP, it will be visible
when starting nr-softmodem.
I have to make both together, since they both depend on
AnalyzeBuildLogs().
- remove mkdir for legacy sub-logfiles in build_oai, that does not exist
anymore; also, remove corresponding log analysis, I think we don't use
it and it is overkill (either build works or not)
- manually try to copy "all.txt"
- explicitly list log files to copy, then analyze
- AnalyzeBuildLogs(): pass log files to inspect explicitly
Set the context for each task step. Set the artifact result path based
on the XML, so it is harmonized; currently, most pipelines use the "YAML
path" of docker-compose, but that does not always make sense. For
instance, Ping/Iperf() needs to look up this path, but does not
inherently belong to one specific YAML path (there might even be more).
This class provides a generic "context" for individual tasks about the
current task step, and where to log results. It's used by
archiveArtifact() that would automatically log files in the right
directory, based on a directory in the context, and in a way that order
of files is preserved for long-running pipelines.
Note: this does not actually work. In the pipelines, we see that for
some reason, quectel-CM cannot establish a connection. Manually, it
works though(?)
Robert Schmidt