maki/openairinterface5g
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/phy_simulators_ctest' into integration_2025_w20 (!3379)

Browse Source 
Add physim tests into ctest framework

Convert physim testcases listed in
cmake_targets/autotests/test_case_list.xml into ctest. Note: Tests for
dlsim_tm4 are excluded, due to compilation issues.

CTest evaluates success based on return code: '0' = pass, non-zero =
fail. For some simulators, the return code is non-zero even when the
test is successful (as indicated by success messages in the test logs).
For some simulators, the return code is always '0'. This inconsistency
has been address in the first two commits of this MR.

To build and execute these tests:

- using cmake

    cd openairinterface5g
    mkdir build && cd build # you can also do that in cmake_targets/ran_build/build
    cmake .. -GNinja -DENABLE_PHYSIM_TESTS=ON && ninja tests
    ctest

- using build_oai: physim-related ctests are activated when PHY
  simulators are built (use build_oai with '-P' or '--phy_simulators'
  option)

    cd openairinterface5g/cmake_targets
    ./build_oai --ninja --phy_simulators
    cd ran_build/build
    ctest

Link to newly created 4g and 5g RAN-Physim-Cluster pipelines:

- RAN-Physim-Cluster-4G: https://jenkins-oai.eurecom.fr/job/RAN-Physim-Cluster-4G/
- RAN-Physim-Cluster-5G: https://jenkins-oai.eurecom.fr/job/RAN-Physim-Cluster-5G/
This commit is contained in:
Robert Schmidt
2025-05-17 12:05:27 +02:00
parent 36ee23255c c861dcb7d0
commit 918ea49686
126 changed files with 782 additions and 5539 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/physical-simulators.md
+137 -111
View File
@@ -1,135 +1,161 @@
This document describes the physical simulators, also called "unitary
simulators", or short "physims".
# Physical Simulators ("Physims") User Guide
This document provides an overview and usage guide for the **physical simulators**, also referred to as **unitary
simulators** or simply **physims**, used in the OpenAirInterface (OAI) project for LTE and 5G PHY layer validation.
[[_TOC_]]
# Introduction
There are several unitary simulations for the physical layer, testing
individual L1 transport/control channels using Monte-Carlo simulations, for
instance:
**Unitary simulations** are standalone test programs designed to validate individual physical layer (L1) transport or
control channels. These simulations are based on **Monte Carlo techniques**, enabling statistical evaluation of
performance metrics such as block error rate (BLER), hybrid automatic repeat request (HARQ) throughput, and decoding
accuracy under various signal conditions.
- `nr_pucchsim` for 5G PUCCH,
- `nr_dlsim` for 5G DLSCH/PDSCH,
- `nr_prachsim` for 5G PRACH,
- `dlsim` for 4G DLSCH/PDSCH, etc.
Physims are essential for:
These simulators constitute the starting point for testing any new code in the
PHY layer, and are required tests to be performed before committing any new
code to the repository, in the sense that all of these should compile and
execute correctly. They provide estimates of the transport block error rates
and HARQ thoughput, DCI error rates, etc. The simulators reside in
`openair1/SIMULATION/LTE_PHY` and `openair1/SIMULATION/NR_PHY`. For instance,
`nr_dlsim` is built from `openair1/SIMULATION/NR_PHY/dlsim.c`.
* Debugging and evaluating new PHY code in isolation
* Regression testing
* Ensuring correctness before merging new contributions into the repository
Further unitary simulation of the coding subsystem components also exists in
`openair1/PHY/CODING/TESTBENCH`, e.g., `ldpctest` for encoding/decoding of
LDPC.
## Examples of Simulators
These are the simulators known to work properly and tested in CI:
- 4G: `dlsim`, `ulsim`
- 5G: `nr_dlsim`, `nr_ulsim`, `nr_dlschsim`, `nr_ulschsim`, `ldpctest`,
`nr_pbchsim`, `nr_prachsim`, `nr_pucchsim`, `polartest`, `smallblocktest`,
`nr_psbchsim` (sidelink)
| Technology | Simulators | Description |
|------------|-------------------------------------------|----------------------------------|
| 4G LTE | `dlsim`, `ulsim` | Downlink and uplink simulators |
| 5G NR | `nr_dlsim`, `nr_ulsim` | Downlink and uplink simulators |
| | `nr_dlschsim`, `nr_ulschsim` | HARQ and TB throughput tests |
| | `nr_pucchsim` | Control channel simulation |
| | `nr_pbchsim` | Broadcast channel simulation |
| | `nr_prachsim` | PRACH simulation |
| | `nr_psbchsim` | Sidelink simulation |
| Coding | `ldpctest`, `polartest`, `smallblocktest` | LDPC, Polar, and other FEC tests |
# How to run
### Source Locations
You first have to build simulators:
* 4G PHY simulators: `openair1/SIMULATION/LTE_PHY/`
* 5G PHY simulators: `openair1/SIMULATION/NR_PHY/`
* Coding unit tests: `openair1/PHY/CODING/TESTBENCH/`
```bash
cd openairinterface5g/
. oaienv # important for the run_exec_autotests.bash script
cd cmake_targets/
./build_oai --ninja -P
```
This builds the simulators in the default build directory `ran_build/build`
(all paths given here are relative to `cmake_targets/` in the root of
`openairinterface5g/`). To run all simulators in this default build directory
of above `build_oai` invocation, run the `run_exec_autotests.bash` script:
```bash
autotests/run_exec_autotests.bash
```
This runs all simulations as defined in `autotests/test_case_list.xml`.
To run a group of tests, you can specify the group using `-g`. You can also
specify the build directory, after building the simulators, which, for the
default directory, is:
```bash
autotests/run_exec_autotests.bash -g "nr_dlsim.basic" -d ran_build/build/
```
This will run the `nr_dlsim.basic` group, as listed in
`autotests/test_case_list.xml`.
## How to interpret the output
From the output, you can see the invoked tests, its parameters, and the log files.
Example:
```
[user@oai cmake_targets]$ autotests/run_exec_autotests.bash -g "nr_dlsim.basic" -d ran_build/build/
[...]
Test case nr_dlsim.basic match found in group
name = nr_dlsim.basic
Description = nr_dlsim Test cases. (Test1: 106 PRB),
(Test2: 217 PRB),
(Test3: 273 PRB),
(Test4: HARQ test 25% TP 4 rounds),
(Test5: HARQ test 33% TP 3 rounds),
(Test6: HARQ test 50% TP 2 rounds),
(Test7: 25 PRBs, 15 kHz SCS)
main_exec = ran_build/build//nr_dlsim
main_exec_args = -n100 -R106 -b106 -s5
-n100 -R217 -b217 -s5
-n100 -R273 -b273 -s5
-n100 -s1 -S2 -t25
-n100 -s1 -S2 -t33
-n100 -s5 -S7 -t50
-n100 -m0 -e0 -R25 -b25 -i 2 1 0
search_expr_true = PDSCH test OK
search_expr_false = segmentation fault|assertion|exiting|fatal
tags = test1 test2 test3 test4 test5 test6 test7
nruns = 3
i = PDSCH test OK
Test1: 106 PRB
Executing test case nr_dlsim.basic.test1, Run Index = 1, Execution Log file = /home/oai/openairinterface5g/cmake_targets/autotests/log/nr_dlsim.basic/test.nr_dlsim.basic.test1.run_1
Executing test case nr_dlsim.basic.test1, Run Index = 2, Execution Log file = /home/oai/openairinterface5g/cmake_targets/autotests/log/nr_dlsim.basic/test.nr_dlsim.basic.test1.run_2
Executing test case nr_dlsim.basic.test1, Run Index = 3, Execution Log file = /home/oai/openairinterface5g/cmake_targets/autotests/log/nr_dlsim.basic/test.nr_dlsim.basic.test1.run_3
[...]
nr_dlsim.basic.test1 RUN = 1 Result = PASS
nr_dlsim.basic.test1 RUN = 2 Result = PASS
nr_dlsim.basic.test1 RUN = 3 Result = PASS
execution nr_dlsim.basic.test1 {Test1: 106 PRB} Run_Result = " Run_1 =PASS Run_2 =PASS Run_3 =PASS" Result = PASS
```bash
# 5G Downlink simulator
openair1/SIMULATION/NR_PHY/dlsim.c
```
You see:
- the `Description` of the test cases (here, 7 tests)
- the executable that is used, and the parameters (`main_exec` and
`main_exec_args`), for instance `-n100 -R106 -b106 -s5` for `Test1`
- what is the condition for a successful test
- where are the log files
- if the tests passed or not `PASS`, otherwise `FAIL` and a summary for the
three runs.
# How to Run Simulators Using `ctest`
## How to change the parameters
You can modify the parameters in `autotests/test_case_list.xml`. To learn about
their meaning, run the simulator manually with flag `-h` ("help"). For
instance, for the above run, it would be
## Option 1: Using CMake
```bash
ran_build/build/nr_dlsim -h
cd openairinterface5g
mkdir build && cd build
cmake .. -GNinja -DENABLE_PHYSIM_TESTS=ON
ninja tests
ctest
```
## How to change the code
## Option 2: Using the `build_oai` script
You can modify the source files to adapt to your needs. For instance, for
`nr_dlsim` above, refer to file `openair1/SIMULATION/NR_PHY/dlsim.c`.
This method simplifies the process by automatically building the simulators with ctest support.
# CI
```bash
cd openairinterface5g/cmake_targets
./build_oai --ninja --phy_simulators
cd ran_build/build
ctest
```
These tests are run in this pipeline:
[RAN-PhySim-Cluster](https://jenkins-oai.eurecom.fr/job/RAN-PhySim-Cluster/).
# `ctest` Usage Tips
## Useful Options
Use the following options to customize test execution:
| Option | Description |
|------------------|------------------------------------------------------|
| `-R <regex>` | Run tests matching the regex pattern (by name) |
| `-L <regex>` | Run tests with labels matching the regex pattern |
| `-E <regex>` | Exclude tests matching the regex pattern (by name) |
| `-LE <regex>` | Exclude tests with labels matching the regex pattern |
| `--print-labels` | Display all available test labels |
| `-j <jobs>` | Run tests in parallel using specified number of jobs |
For the complete list of `ctest` options, refer to the manual:
```bash
man ctest
```
## Example
```bash
# Run only NR ULSCH simulator tests, in parallel
ctest -L nr_ulsch -j 4
```
# Adding a New Physim Test
To define a new test or modify existing ones, update the following file:
```
openair1/PHY/CODING/tests/CMakeLists.txt
```
Use the `add_physim_test` macro with the following arguments:
```cmake
add_physim_test(<test_gen> <test_exec> <test_description> <test_label> <test_cl_options>)
```
### Arguments:
* `<test_gen>`: Test generation (e.g., `4g` or `5g`)
* `<test_exec>`: Name of the test executable (e.g., `nr_prachsim`)
* `<test_description>`: Description shown in `ctest` output, useful for categorization and indexing.
* `<test_label>`: Label used for filtering tests (via `-L`), shown in the ctest output summary as a descriptive tag.
* `<test_cl_options>`: Command line options passed to the test
### Example:
```cmake
# add_physim_test(<test_gen> <test_exec> <test_description> <test_label> <test_cl_options>)
add_physim_test(5g nr_prachsim test8 "15kHz SCS, 25 PRBs" -a -s -30 -n 300 -p 99 -R 25 -m 0)
```
These tests are run automatically as part of the following
pipelines: [RAN-PhySim-Cluster-4G](https://jenkins-oai.eurecom.fr/job/RAN-PhySim-Cluster-4G/) and [RAN-PhySim-Cluster-5G](https://jenkins-oai.eurecom.fr/job/RAN-PhySim-Cluster-5G/)
### How to rerun failed CI tests using `ctest`
Ctest automatically logs the failed tests in LastTestsFailed.log. This log is archived in
the CI artifacts and can be reused locally to rerun only those failed tests.
```bash
# 1. Navigate to the build directory, build physims
cd ~/openairinterface5g/build
cmake .. -GNinja -DENABLE_PHYSIM_TESTS=ON && ninja tests
# 2. Create ctest directory structure
ctest -N --quiet
# 3. Unzip the test logs artifact from the CI run
unzip /path/to/test_logs_123.zip
# 4. Copy the LastTestsFailed.log file to the expected location
cp /path/to/test_logs_123/test_logs/LastTestsFailed.log ./Testing/Temporary/
# 5. Rerun only the failed tests using ctest
ctest --rerun-failed
```
# Legacy Bash Autotest (Deprecated)
> **Note:** Autotest script, configuration, and documentation, are no longer maintained.
For legacy support or archival purposes, you can still find this implementation by checking out the historical tag:
```bash
git checkout 2025.w18
```