From 03350ab21b118f2794667eea11f2b398a3ecb0cb Mon Sep 17 00:00:00 2001 From: Romain Beurdouche Date: Thu, 19 Feb 2026 09:57:08 +0000 Subject: [PATCH] doc(tuning_and_security): extend the performance tuning section Extend the performance tuning section of documentation to include explanation on the `--L1s.L1_num_tx_sym_per_thread` option and on the AMD EPYC core complex behavior --- doc/NR_SA_Tutorial_COTS_UE.md | 1 + doc/tuning_and_security.md | 103 ++++++++++++++++++++++++++++++++++ 2 files changed, 104 insertions(+) diff --git a/doc/NR_SA_Tutorial_COTS_UE.md b/doc/NR_SA_Tutorial_COTS_UE.md index 7f73be72f0..b6b8d4e235 100644 --- a/doc/NR_SA_Tutorial_COTS_UE.md +++ b/doc/NR_SA_Tutorial_COTS_UE.md @@ -193,6 +193,7 @@ sudo ethtool -G enp1s0f0 tx 4096 rx 4096 - Enable Performance Mode `sudo cpupower idle-set -D 0` - If you get real-time problems on heavy UL traffic, reduce the maximum UL MCS using an additional command-line switch: `--MACRLCs.[0].ul_max_mcs 14`. - You can also reduce the number of LDPC decoder iterations, which will make the LDPC decoder take less time: `--L1s.[0].max_ldpc_iterations 4`. +- Read more on system tuning in the [dedicated document on performance tuning](./tuning_and_security.md#performance-tuning). ### 6.3 Uplink issues related with noise on the DC carriers diff --git a/doc/tuning_and_security.md b/doc/tuning_and_security.md index af8e77c0f6..11e07b6e9b 100644 --- a/doc/tuning_and_security.md +++ b/doc/tuning_and_security.md @@ -69,6 +69,109 @@ sudo sysctl -n -e -q -w net.core.wmem_default=134217728 sudo sysctl -n -e -q -w net.core.wmem_max=134217728 ``` +### System tuning +In order to get an optimal real-time behavior, a few tunings can be performed on the host system: +- The use of isolated cores for the softmodem prevents competitions on the usage of core between the softmodem and other processes. + Core isolation is enabled through the kernel command line. + **Warning: modifying the kernel command line can harm the OS behavior. Proceed with caution.** + Refer to the [OAI 7.2 Fronthaul Interface Tutorial](./ORAN_FHI7.2_Tutorial.md) for examples. + +### Softmodem tuning +The way the NR softmodem uses the computing ressource can be configured. +It can have a significant effect on the performance and real-time behavior: +- The L1 TX and L1 RX threads are the two main threads + executing the L1 RX and L1 TX pipelines. + These threads are ideally assigned to two dedicated cores. + To be dedicated, the cores should be isolated in the + kernel parameters and not be assigned elsewhere. + They can be assigned to specified cores with options + `--L1s.[0].L1_tx_thread_core` and `--L1s.[0].L1_rx_thread_core` + followed by a core id. +- The thread pool is a group of processor cores over which + some baseband processing worker cores execute. + It is configured by providing a list of core ids + after option `--thread-pool`. + `-1` can also be passed instead of a core id + in order to use a floating core. + By default, the thread pool is 8 floating cores. +- PDSCH generation (i.e., layer mapping and precoding) is by default executed + in the L1 TX thread but can be multithreaded using the thread pool. + This is enabled by option `--L1s.[0].L1_num_tx_sym_per_thread` followed + by the number of symbols that should be processed in each thread. + This option can also be set in the gNB configuration file in field + `L1_num_tx_sym_per_thread` in the `L1s` section. + +### Workarounds +If the real-time performance remains bad after tuning the system and softmodem, +some workarounds allow to lower the computing demand +at the cost of lower network performance: +- If you get real-time problems on heavy UL traffic, + reduce the maximum UL MCS using an additional + command-line switch: `--MACRLCs.[0].ul_max_mcs 14`. + This comes at the cost of a lower spectral efficiency + (i.e., less data for the same radio resource). +- You can also reduce the number of LDPC decoder iterations, + which will make the LDPC decoder take less time: + `--L1s.[0].max_ldpc_iterations 4`. + The default number of LDPC iterations is 8. + Lowering the number of iteration comes at the cost + of more unsuccessful transmissions. + OAI offers multiple implementation of LDPC coding, + including offloading to an accelerator, + the number of LDPC iteration should be chosen accordingly. + +### Known hardware behaviors +Here is a **non-exhaustive** list of known behaviors related to hardware architecture: +- On some AMD EPYC series processors with Zen architecture + (at least every Zen4, Zen4c, Zen5 and Zen5c based processors + experience this behavior), + the processor is made of multiple dies holding one or multiple + core complexes which are groups of cores with an L3 cache. + This means that cores from different core complexes + do not share the same L3 cache and communication between + these cores implies inter L3 cache communication + within a die or, even worse, between dies, + which has a cost in term of latency. + Depending on the system configuration, + the NUMA topology may reflect this physical topology, + which can induce even further latency for + inter core complex communication. + The softmodem is sensitive to this latency and its performance + can be harmed if it uses cores across the border of dies or core complexes, + especially when multithreading of PDSCH generation is enabled + (argument of `--L1s.[0].L1_num_tx_sym_per_thread` is superior to 0). + **Solution**: We recommend to use only one core complex + or one die for allocating cores to nr-softmodem process. +- NUMA architecture: Make sure you don't assign cores to nr-softmodem from + different numa nodes. Cores from different numa cores can induce latency. + It is preferred to use the cores from numa node + which is used by the Fronthaul NIC. + +**Example**: How to assign cores to `nr-softmodem` process for gNB/DU connected to an o-ran 7.2 O-RU on EPYC 9575F (64 Zen5) +assuming that we have configured `L1s.[0].L1_num_tx_sym_per_thread = 1` + +There are 8 cores per core complex and one core complex per die. You can visulize this topology using + +```bash +# Ubuntu 25.04 +for X in $(seq 0 63); do echo -n "cpu$X -> die "; cat /sys/devices/system/cpu/cpu$X/topology/die_id; done +``` + +A 100MHz 4x4 FR1 gNB/DU with FHI 7.2 can be executed with full capabilities (4DL and 2UL layers) +on a single core complex (cpus 0-7 in this example) by folowing the instructions of the +[OAI 7.2 Fronthaul Interface Tutorial](./ORAN_FHI7.2_Tutorial.md) +with the following core assignment: + +- `L1s.[0].L1_tx_thread_core = 0` +- `L1s.[0].L1_rx_thread_core = 1` +- `RUs.[0].ru_thread_core = 2` +- `fhi_72.system_core = 3` +- `fhi_72.io_core = 4` +- `fhi_72.worker_cores.[0] = 5` (minimum 1 core) +- `thread-pool: 2,3,6,7` + (thread pool can overlap with `RUs.[0].ru_thread_core` and `fhi_72.system_core`) + + ## Capabilities Historically, all softmodems are executed as `root`, typically using `sudo`.