diff --git a/doc/NR_SA_Tutorial_COTS_UE.md b/doc/NR_SA_Tutorial_COTS_UE.md index 7f73be72f0..b6b8d4e235 100644 --- a/doc/NR_SA_Tutorial_COTS_UE.md +++ b/doc/NR_SA_Tutorial_COTS_UE.md @@ -193,6 +193,7 @@ sudo ethtool -G enp1s0f0 tx 4096 rx 4096 - Enable Performance Mode `sudo cpupower idle-set -D 0` - If you get real-time problems on heavy UL traffic, reduce the maximum UL MCS using an additional command-line switch: `--MACRLCs.[0].ul_max_mcs 14`. - You can also reduce the number of LDPC decoder iterations, which will make the LDPC decoder take less time: `--L1s.[0].max_ldpc_iterations 4`. +- Read more on system tuning in the [dedicated document on performance tuning](./tuning_and_security.md#performance-tuning). ### 6.3 Uplink issues related with noise on the DC carriers diff --git a/doc/tuning_and_security.md b/doc/tuning_and_security.md index af8e77c0f6..11e07b6e9b 100644 --- a/doc/tuning_and_security.md +++ b/doc/tuning_and_security.md @@ -69,6 +69,109 @@ sudo sysctl -n -e -q -w net.core.wmem_default=134217728 sudo sysctl -n -e -q -w net.core.wmem_max=134217728 ``` +### System tuning +In order to get an optimal real-time behavior, a few tunings can be performed on the host system: +- The use of isolated cores for the softmodem prevents competitions on the usage of core between the softmodem and other processes. + Core isolation is enabled through the kernel command line. + **Warning: modifying the kernel command line can harm the OS behavior. Proceed with caution.** + Refer to the [OAI 7.2 Fronthaul Interface Tutorial](./ORAN_FHI7.2_Tutorial.md) for examples. + +### Softmodem tuning +The way the NR softmodem uses the computing ressource can be configured. +It can have a significant effect on the performance and real-time behavior: +- The L1 TX and L1 RX threads are the two main threads + executing the L1 RX and L1 TX pipelines. + These threads are ideally assigned to two dedicated cores. + To be dedicated, the cores should be isolated in the + kernel parameters and not be assigned elsewhere. + They can be assigned to specified cores with options + `--L1s.[0].L1_tx_thread_core` and `--L1s.[0].L1_rx_thread_core` + followed by a core id. +- The thread pool is a group of processor cores over which + some baseband processing worker cores execute. + It is configured by providing a list of core ids + after option `--thread-pool`. + `-1` can also be passed instead of a core id + in order to use a floating core. + By default, the thread pool is 8 floating cores. +- PDSCH generation (i.e., layer mapping and precoding) is by default executed + in the L1 TX thread but can be multithreaded using the thread pool. + This is enabled by option `--L1s.[0].L1_num_tx_sym_per_thread` followed + by the number of symbols that should be processed in each thread. + This option can also be set in the gNB configuration file in field + `L1_num_tx_sym_per_thread` in the `L1s` section. + +### Workarounds +If the real-time performance remains bad after tuning the system and softmodem, +some workarounds allow to lower the computing demand +at the cost of lower network performance: +- If you get real-time problems on heavy UL traffic, + reduce the maximum UL MCS using an additional + command-line switch: `--MACRLCs.[0].ul_max_mcs 14`. + This comes at the cost of a lower spectral efficiency + (i.e., less data for the same radio resource). +- You can also reduce the number of LDPC decoder iterations, + which will make the LDPC decoder take less time: + `--L1s.[0].max_ldpc_iterations 4`. + The default number of LDPC iterations is 8. + Lowering the number of iteration comes at the cost + of more unsuccessful transmissions. + OAI offers multiple implementation of LDPC coding, + including offloading to an accelerator, + the number of LDPC iteration should be chosen accordingly. + +### Known hardware behaviors +Here is a **non-exhaustive** list of known behaviors related to hardware architecture: +- On some AMD EPYC series processors with Zen architecture + (at least every Zen4, Zen4c, Zen5 and Zen5c based processors + experience this behavior), + the processor is made of multiple dies holding one or multiple + core complexes which are groups of cores with an L3 cache. + This means that cores from different core complexes + do not share the same L3 cache and communication between + these cores implies inter L3 cache communication + within a die or, even worse, between dies, + which has a cost in term of latency. + Depending on the system configuration, + the NUMA topology may reflect this physical topology, + which can induce even further latency for + inter core complex communication. + The softmodem is sensitive to this latency and its performance + can be harmed if it uses cores across the border of dies or core complexes, + especially when multithreading of PDSCH generation is enabled + (argument of `--L1s.[0].L1_num_tx_sym_per_thread` is superior to 0). + **Solution**: We recommend to use only one core complex + or one die for allocating cores to nr-softmodem process. +- NUMA architecture: Make sure you don't assign cores to nr-softmodem from + different numa nodes. Cores from different numa cores can induce latency. + It is preferred to use the cores from numa node + which is used by the Fronthaul NIC. + +**Example**: How to assign cores to `nr-softmodem` process for gNB/DU connected to an o-ran 7.2 O-RU on EPYC 9575F (64 Zen5) +assuming that we have configured `L1s.[0].L1_num_tx_sym_per_thread = 1` + +There are 8 cores per core complex and one core complex per die. You can visulize this topology using + +```bash +# Ubuntu 25.04 +for X in $(seq 0 63); do echo -n "cpu$X -> die "; cat /sys/devices/system/cpu/cpu$X/topology/die_id; done +``` + +A 100MHz 4x4 FR1 gNB/DU with FHI 7.2 can be executed with full capabilities (4DL and 2UL layers) +on a single core complex (cpus 0-7 in this example) by folowing the instructions of the +[OAI 7.2 Fronthaul Interface Tutorial](./ORAN_FHI7.2_Tutorial.md) +with the following core assignment: + +- `L1s.[0].L1_tx_thread_core = 0` +- `L1s.[0].L1_rx_thread_core = 1` +- `RUs.[0].ru_thread_core = 2` +- `fhi_72.system_core = 3` +- `fhi_72.io_core = 4` +- `fhi_72.worker_cores.[0] = 5` (minimum 1 core) +- `thread-pool: 2,3,6,7` + (thread pool can overlap with `RUs.[0].ru_thread_core` and `fhi_72.system_core`) + + ## Capabilities Historically, all softmodems are executed as `root`, typically using `sudo`. diff --git a/executables/nr-gnb.c b/executables/nr-gnb.c index 74b6835e3f..abbfa05df1 100644 --- a/executables/nr-gnb.c +++ b/executables/nr-gnb.c @@ -208,8 +208,7 @@ static size_t dump_L1_meas_stats(PHY_VARS_gNB *gNB, RU_t *ru, char *output, size output += print_meas_log(&gNB->dlsch_encoding_stats, "DLSCH encoding", NULL, NULL, output, end - output); output += print_meas_log(&gNB->dlsch_scrambling_stats, "DLSCH scrambling", NULL, NULL, output, end-output); output += print_meas_log(&gNB->dlsch_modulation_stats, "DLSCH modulation", NULL, NULL, output, end - output); - output += print_meas_log(&gNB->dlsch_resource_mapping_stats, "DLSCH resource mapping", NULL, NULL, output,end-output); - output += print_meas_log(&gNB->dlsch_precoding_stats, "DLSCH precoding", NULL, NULL, output,end-output); + output += print_meas_log(&gNB->dlsch_pdsch_generation_stats, "PDSCH generation", NULL, NULL, output, end - output); output += print_meas_log(&gNB->phy_proc_rx, "L1 Rx processing", NULL, NULL, output, end - output); output += print_meas_log(&gNB->ts_deinterleave, "UL segment deinterleaving", NULL, NULL, output, end - output); output += print_meas_log(&gNB->ts_rate_unmatch, "UL segment rate recovery", NULL, NULL, output, end - output); @@ -280,8 +279,7 @@ void *nrL1_stats_thread(void *param) { reset_meas(&gNB->rx_pusch_stats); reset_meas(&gNB->dlsch_scrambling_stats); reset_meas(&gNB->dlsch_modulation_stats); - reset_meas(&gNB->dlsch_resource_mapping_stats); - reset_meas(&gNB->dlsch_precoding_stats); + reset_meas(&gNB->dlsch_pdsch_generation_stats); while (!oai_exit) { sleep(1); if (ftruncate(fileno(fd), 0) != 0 || fseek(fd, 0, SEEK_SET) != 0) { @@ -309,8 +307,6 @@ void init_gNB_Tpool(int inst) PHY_VARS_gNB *gNB; gNB = RC.gNB[inst]; gNB_L1_proc_t *proc = &gNB->proc; - // PUSCH symbols per thread need to be calculated by how many threads we have - gNB->num_pusch_symbols_per_thread = 1; // ULSCH decoding threadpool initTpool(get_softmodem_params()->threadPoolConfig, &gNB->threadPool, cpumeas(CPUMEAS_GETSTATE)); diff --git a/openair1/PHY/NR_TRANSPORT/nr_dlsch.c b/openair1/PHY/NR_TRANSPORT/nr_dlsch.c index 73ba4be4af..d227148353 100644 --- a/openair1/PHY/NR_TRANSPORT/nr_dlsch.c +++ b/openair1/PHY/NR_TRANSPORT/nr_dlsch.c @@ -17,8 +17,6 @@ #include "executables/softmodem-common.h" #include "SCHED_NR/sched_nr.h" -// #define DEBUG_DLSCH -// #define DEBUG_DLSCH_MAPPING #include #define USE128BIT @@ -46,21 +44,9 @@ static int do_ptrs_symbol(const nfapi_nr_dl_tti_pdsch_pdu_rel15_t *rel15, /* check if cuurent RE is PTRS RE*/ uint16_t beta_ptrs = 1; txF[k] = c16mulRealShift(mod_ptrs[ptrs_idx], beta_ptrs * amp, 15); -#ifdef DEBUG_DLSCH_MAPPING - printf("ptrs_idx %d\t \t k %d \t \t txdataF: %d %d, mod_ptrs: %d %d\n", - ptrs_idx, - k, - txF[k].r, - txF[k].i, - mod_ptrs[ptrs_idx].r, - mod_ptrs[ptrs_idx].i); -#endif ptrs_idx++; } else { txF[k] = c16mulRealShift(*in++, amp, 15); -#ifdef DEBUG_DLSCH_MAPPING - printf("k %d \t txdataF: %d %d\n", k, txF[k].r, txF[k].i); -#endif } k++; } @@ -74,9 +60,6 @@ typedef union { static inline int interleave_with_0_signal_first(c16_t *output, c16_t *mod_dmrs, const int16_t amp_dmrs, int sz) { -#ifdef DEBUG_DLSCH_MAPPING - printf("doing DMRS pattern for port 0 : d0 0 d1 0 ... dNm2 0 dNm1 0 (ul %d, rr %d)\n", upper_limit, remaining_re); -#endif // add filler to process all as SIMD c16_t *out = output; int i = 0; @@ -126,9 +109,6 @@ static inline int interleave_with_0_signal_first(c16_t *output, c16_t *mod_dmrs, static inline int interleave_with_0_start_with_0(c16_t *output, c16_t *mod_dmrs, const int16_t amp_dmrs, int sz) { -#ifdef DEBUG_DLSCH_MAPPING - printf("doing DMRS pattern for port 2 : 0 d0 0 d1 ... 0 dNm2 0 dNm1\n"); -#endif c16_t *out = output; int i = 0; int end = sz / 2; @@ -177,9 +157,6 @@ static inline int interleave_with_0_start_with_0(c16_t *output, c16_t *mod_dmrs, static inline int interleave_signals(c16_t *output, c16_t *signal1, const int amp, c16_t *signal2, const int amp2, int sz) { -#ifdef DEBUG_DLSCH_MAPPING - printf("doing DMRS pattern for port 0 : d0 X0 d1 X1 ... dNm2 XNm2 dNm1 XNm1\n"); -#endif // add filler to process all as SIMD c16_t *out = output; int i = 0; @@ -311,22 +288,22 @@ static inline void neg_dmrs(c16_t *in, c16_t *out, int sz) *out++ = i % 2 ? (c16_t){-in[i].r, -in[i].i} : in[i]; } -static inline int do_onelayer(NR_DL_FRAME_PARMS *frame_parms, - int slot, - const nfapi_nr_dl_tti_pdsch_pdu_rel15_t *rel15, - int layer, - c16_t *output, - c16_t *txl_start, - int start_sc, - int symbol_sz, - int l_symbol, - uint16_t dlPtrsSymPos, - int n_ptrs, - int amp, - int16_t amp_dmrs, - int l_prime, - nfapi_nr_dmrs_type_e dmrs_Type, - c16_t *dmrs_start) +static inline void do_onelayer(NR_DL_FRAME_PARMS *frame_parms, + int slot, + const nfapi_nr_dl_tti_pdsch_pdu_rel15_t *rel15, + int layer, + c16_t *output, + c16_t *txl_start, + int start_sc, + int symbol_sz, + int l_symbol, + uint16_t dlPtrsSymPos, + int n_ptrs, + int amp, + int16_t amp_dmrs, + int l_prime, + nfapi_nr_dmrs_type_e dmrs_Type, + c16_t *dmrs_start) { c16_t *txl = txl_start; const uint sz = rel15->rbSize * NR_NB_SC_PER_RB; @@ -408,7 +385,7 @@ static inline int do_onelayer(NR_DL_FRAME_PARMS *frame_parms, } else { // no PTRS or DMRS in this symbol txl += no_ptrs_dmrs_case(output + start_sc, txl, amp, sz); } // no DMRS/PTRS in symbol - return txl - txl_start; + return; } static inline void do_txdataF(c16_t **txdataF, @@ -472,28 +449,123 @@ static inline void do_txdataF(c16_t **txdataF, } // RB loop: while(rb < rel15->rbSize) } +typedef struct pdschSymbolProc_s { + PHY_VARS_gNB *gNB; + NR_DL_FRAME_PARMS *frame_parms; + const nfapi_nr_dl_tti_pdsch_pdu_rel15_t *rel15; + unsigned int slot; + unsigned int startSymbol; + unsigned int numSymbols; + task_ans_t *ans; + unsigned int layerSz2; + unsigned int dlPtrsSymPos; + unsigned int n_ptrs; + unsigned int beam_nb; + unsigned int re_beginning_of_symbol[14]; + c16_t *tx_layers[4]; + time_stats_t dlsch_resource_mapping_stats; + time_stats_t dlsch_precoding_stats; +} pdschSymbolProc_t; + +static void nr_pdsch_symbol_processing(void *arg) +{ + pdschSymbolProc_t *rdata = (pdschSymbolProc_t *)arg; + + PHY_VARS_gNB *gNB = rdata->gNB; + NR_DL_FRAME_PARMS *frame_parms = rdata->frame_parms; + const nfapi_nr_dl_tti_pdsch_pdu_rel15_t *rel15 = rdata->rel15; + int slot = rdata->slot; + c16_t *tx_layers[rel15->nrOfLayers]; + for (int l = 0; l < rel15->nrOfLayers; l++) + tx_layers[l] = rdata->tx_layers[l]; + const int nb_re_dmrs = rel15->numDmrsCdmGrpsNoData * (rel15->dmrsConfigType == NFAPI_NR_DMRS_TYPE1 ? 6 : 4); + const int n_dmrs = (rel15->BWPStart + rel15->rbStart + rel15->rbSize) * nb_re_dmrs; + // Loop Over OFDM symbols: + c16_t mod_dmrs[(n_dmrs + 63) & ~63] __attribute__((aligned(64))); + const int symbol_sz = frame_parms->ofdm_symbol_size; + + c16_t **txdataF = gNB->common_vars.txdataF[rdata->beam_nb]; + uint16_t start_sc = (rel15->rbStart + rel15->BWPStart) * NR_NB_SC_PER_RB; + + for (int l_symbol = rdata->startSymbol; l_symbol < rdata->startSymbol + rdata->numSymbols; l_symbol++) { + start_meas(&rdata->dlsch_resource_mapping_stats); + int l_prime = 0; // single symbol layer 0 + int l_overline = get_l0(rel15->dlDmrsSymbPos); + + /// DMRS QPSK modulation + if ((rel15->dlDmrsSymbPos & (1 << l_symbol))) { // DMRS time occasion + // The reference point for is subcarrier -1 of the lowest-numbered resource block in CORESET 0 if the corresponding + // PDCCH is associated with CORESET -1 and Type0-PDCCH common search space and is addressed to SI-RNTI + // 2GPP TS 38.211 V15.8.0 Section 7.4.1.1.2 Mapping to physical resources + if (l_symbol == (l_overline + 1)) // take into account the double DMRS symbols + l_prime = 1; + else if (l_symbol > (l_overline + 1)) { // new DMRS pair + l_overline = l_symbol; + l_prime = 0; + } + const uint32_t *gold = nr_gold_pdsch(frame_parms->N_RB_DL, + frame_parms->symbols_per_slot, + rel15->dlDmrsScramblingId, + rel15->SCID, + slot, + l_symbol); + // Qm = 1 as DMRS is QPSK modulated + nr_modulation(gold, n_dmrs * DMRS_MOD_ORDER, DMRS_MOD_ORDER, (int16_t *)mod_dmrs); + + } + uint32_t dmrs_idx = rel15->rbStart; + if (rel15->refPoint == 0) + dmrs_idx += rel15->BWPStart; + dmrs_idx *= rel15->dmrsConfigType == NFAPI_NR_DMRS_TYPE1 ? 6 : 4; + c16_t txdataF_precoding[rel15->nrOfLayers][symbol_sz] __attribute__((aligned(64))); + for (int layer = 0; layer < rel15->nrOfLayers; layer++) { + do_onelayer(frame_parms, + slot, + rel15, + layer, + txdataF_precoding[layer], + tx_layers[layer] + rdata->re_beginning_of_symbol[l_symbol], + start_sc, + symbol_sz, + l_symbol, + rdata->dlPtrsSymPos, + rdata->n_ptrs, + gNB->TX_AMP, + min((double)gNB->TX_AMP * sqrt(rel15->numDmrsCdmGrpsNoData), INT16_MAX), + l_prime, + rel15->dmrsConfigType, + mod_dmrs + dmrs_idx); + } // layer loop + stop_meas(&rdata->dlsch_resource_mapping_stats); + + start_meas(&rdata->dlsch_precoding_stats); + for (int ant = 0; ant < frame_parms->nb_antennas_tx; ant++) { + const size_t txdataF_offset_per_symbol = l_symbol * symbol_sz; + do_txdataF(txdataF, symbol_sz, txdataF_precoding, gNB, rel15, ant, start_sc, txdataF_offset_per_symbol); + } + stop_meas(&rdata->dlsch_precoding_stats); + } + // Task running in // completed + completed_task_ans(rdata->ans); +} + static int do_one_dlsch(unsigned char *input_ptr, PHY_VARS_gNB *gNB, NR_gNB_DLSCH_t *dlsch, int slot) { - const int16_t amp = gNB->TX_AMP; NR_DL_FRAME_PARMS *frame_parms = &gNB->frame_parms; time_stats_t *dlsch_scrambling_stats = &gNB->dlsch_scrambling_stats; time_stats_t *dlsch_modulation_stats = &gNB->dlsch_modulation_stats; const nfapi_nr_dl_tti_pdsch_pdu_rel15_t *rel15 = &dlsch->pdsch_pdu->pdsch_pdu_rel15; const int layerSz = frame_parms->N_RB_DL * frame_parms->symbols_per_slot * NR_NB_SC_PER_RB; - const int symbol_sz=frame_parms->ofdm_symbol_size; - const int dmrs_Type = rel15->dmrsConfigType; const int nb_re_dmrs = rel15->numDmrsCdmGrpsNoData * (rel15->dmrsConfigType == NFAPI_NR_DMRS_TYPE1 ? 6 : 4); - const int16_t amp_dmrs = min((double)amp * sqrt(rel15->numDmrsCdmGrpsNoData), INT16_MAX); // 3GPP TS 38.214 Section 4.1: Table 4.1-1 LOG_D(PHY, "pdsch: BWPStart %d, BWPSize %d, rbStart %d, rbsize %d\n", rel15->BWPStart, rel15->BWPSize, rel15->rbStart, rel15->rbSize); - const int n_dmrs = (rel15->BWPStart + rel15->rbStart + rel15->rbSize) * nb_re_dmrs; + const int n_dmrs = rel15->rbSize * nb_re_dmrs; - const int dmrs_symbol_map = rel15->dlDmrsSymbPos; // single DMRS: 010000100 Double DMRS 110001100 const int xOverhead = 0; const int nb_re = (12 * rel15->NrOfSymbols - nb_re_dmrs * get_num_dmrs(rel15->dlDmrsSymbPos) - xOverhead) * rel15->rbSize * rel15->nrOfLayers; @@ -567,21 +639,9 @@ static int do_one_dlsch(unsigned char *input_ptr, PHY_VARS_gNB *gNB, NR_gNB_DLSC start_meas(&gNB->dlsch_pdsch_generation_stats); /// Resource mapping // Non interleaved VRB to PRB mapping - uint16_t start_sc = (rel15->rbStart + rel15->BWPStart) * NR_NB_SC_PER_RB; - -#ifdef DEBUG_DLSCH_MAPPING - printf("PDSCH resource mapping started (start SC %d\tstart symbol %d\tN_PRB %d\tnb_re %d,nb_layers %d)\n", - start_sc, - rel15->StartSymbolIndex, - rel15->rbSize, - nb_re, - rel15->nrOfLayers); -#endif AssertFatal(n_dmrs, "n_dmrs can't be 0\n"); // make a large enough tail to process all re with SIMD regardless a garbadge filler - c16_t mod_dmrs[(n_dmrs+63)&~63] __attribute__((aligned(64))); - unsigned int re_beginning_of_symbol = 0; start_meas(&gNB->dlsch_layer_mapping_stats); int layerSz2 = (layerSz + 63) & ~63; @@ -605,84 +665,69 @@ static int do_one_dlsch(unsigned char *input_ptr, PHY_VARS_gNB *gNB, NR_gNB_DLSC slot, frame_parms->symbols_per_slot, bitmap); - - c16_t **txdataF = gNB->common_vars.txdataF[beam_nb]; stop_meas(&gNB->dlsch_layer_mapping_stats); - // Loop Over OFDM symbols: - for (int l_symbol = rel15->StartSymbolIndex; l_symbol < rel15->StartSymbolIndex + rel15->NrOfSymbols; l_symbol++) { - start_meas(&gNB->dlsch_resource_mapping_stats); - int l_prime = 0; // single symbol layer 0 - int l_overline = get_l0(rel15->dlDmrsSymbPos); -#ifdef DEBUG_DLSCH_MAPPING - printf("PDSCH resource mapping symbol %d\n", l_symbol); -#endif - /// DMRS QPSK modulation - if ((dmrs_symbol_map & (1 << l_symbol))) { // DMRS time occasion - // The reference point for is subcarrier -1 of the lowest-numbered resource block in CORESET 0 if the corresponding - // PDCCH is associated with CORESET -1 and Type0-PDCCH common search space and is addressed to SI-RNTI - // 2GPP TS 38.211 V15.8.0 Section 7.4.1.1.2 Mapping to physical resources - if (l_symbol == (l_overline + 1)) // take into account the double DMRS symbols - l_prime = 1; - else if (l_symbol > (l_overline + 1)) { // new DMRS pair - l_overline = l_symbol; - l_prime = 0; + // spawn symbol threads + + int nb_tasks = 1; + int num_pdsch_symbols_per_task = rel15->NrOfSymbols; + if (gNB->num_pdsch_symbols_per_thread > 0) { + // symbol processing in thread pool enabled + num_pdsch_symbols_per_task = gNB->num_pdsch_symbols_per_thread; + nb_tasks = rel15->NrOfSymbols / num_pdsch_symbols_per_task; + if ((rel15->NrOfSymbols % num_pdsch_symbols_per_task) > 0) + nb_tasks++; + } + pdschSymbolProc_t arr[nb_tasks]; + task_ans_t ans; + init_task_ans(&ans, nb_tasks); + int sz_arr = 0; + unsigned int re_beginning_of_symbol = 0; + int res = 0; + for (int l_symbol = rel15->StartSymbolIndex; l_symbol < rel15->StartSymbolIndex + rel15->NrOfSymbols; + l_symbol += num_pdsch_symbols_per_task) { + pdschSymbolProc_t *rdata = &arr[sz_arr]; + rdata->ans = &ans; + ++sz_arr; + + rdata->gNB = gNB; + rdata->frame_parms = frame_parms; + rdata->rel15 = rel15; + rdata->slot = slot; + rdata->startSymbol = l_symbol; + res = rel15->NrOfSymbols - (l_symbol - rel15->StartSymbolIndex); + if (res >= num_pdsch_symbols_per_task) + rdata->numSymbols = num_pdsch_symbols_per_task; + else + rdata->numSymbols = res; + rdata->layerSz2 = layerSz2; + rdata->dlPtrsSymPos = dlPtrsSymPos; + rdata->n_ptrs = n_ptrs; + rdata->beam_nb = beam_nb; + for (int s = l_symbol; s < l_symbol + rdata->numSymbols; s++) { + rdata->re_beginning_of_symbol[s] = re_beginning_of_symbol; + re_beginning_of_symbol += rel15->rbSize * NR_NB_SC_PER_RB; + if (n_ptrs > 0 && is_ptrs_symbol(s, dlPtrsSymPos)) { + re_beginning_of_symbol -= n_ptrs; + } else if (rel15->dlDmrsSymbPos & (1 << s)) { + re_beginning_of_symbol -= n_dmrs; } -#ifdef DEBUG_DLSCH_MAPPING - printf("dlDmrsScramblingId %d, SCID %d slot %d l_symbol %d\n", rel15->dlDmrsScramblingId, rel15->SCID, slot, l_symbol); -#endif - const uint32_t *gold = nr_gold_pdsch(frame_parms->N_RB_DL, - frame_parms->symbols_per_slot, - rel15->dlDmrsScramblingId, - rel15->SCID, - slot, - l_symbol); - // Qm = 1 as DMRS is QPSK modulated - nr_modulation(gold, n_dmrs * DMRS_MOD_ORDER, DMRS_MOD_ORDER, (int16_t *)mod_dmrs); - -#ifdef DEBUG_DLSCH_MAPPING - printf("DMRS modulation (symbol %d, %d symbols, type %d):\n", l_symbol, n_dmrs, dmrs_Type); - for (int i = 0; i < n_dmrs / 2; i += 8) { - for (int j = 0; j < 8; j++) { - printf("%d %d\t", mod_dmrs[i + j].r, mod_dmrs[i + j].i); - } - printf("\n"); - } -#endif } - uint32_t dmrs_idx = rel15->rbStart; - if (rel15->refPoint == 0) - dmrs_idx += rel15->BWPStart; - dmrs_idx *= dmrs_Type == NFAPI_NR_DMRS_TYPE1 ? 6 : 4; - c16_t txdataF_precoding[rel15->nrOfLayers][symbol_sz] __attribute__((aligned(64))); - int layer_sz = 0; - for (int layer = 0; layer < rel15->nrOfLayers; layer++) { - layer_sz = do_onelayer(frame_parms, - slot, - rel15, - layer, - txdataF_precoding[layer], - tx_layers[layer] + re_beginning_of_symbol, - start_sc, - symbol_sz, - l_symbol, - dlPtrsSymPos, - n_ptrs, - amp, - amp_dmrs, - l_prime, - dmrs_Type, - mod_dmrs + dmrs_idx); - } // layer loop - re_beginning_of_symbol += layer_sz; - stop_meas(&gNB->dlsch_resource_mapping_stats); - - start_meas(&gNB->dlsch_precoding_stats); - for (int ant = 0; ant < frame_parms->nb_antennas_tx; ant++) { - const size_t txdataF_offset_per_symbol = l_symbol * symbol_sz; - do_txdataF(txdataF, symbol_sz, txdataF_precoding, gNB, rel15, ant, start_sc, txdataF_offset_per_symbol); + reset_meas(&rdata->dlsch_resource_mapping_stats); + reset_meas(&rdata->dlsch_precoding_stats); + for (int l = 0; l < rel15->nrOfLayers; l++) + rdata->tx_layers[l] = tx_layers[l]; + if (l_symbol < rel15->StartSymbolIndex + rel15->NrOfSymbols - num_pdsch_symbols_per_task) { + task_t t = {.func = &nr_pdsch_symbol_processing, .args = rdata}; + pushTpool(&gNB->threadPool, t); + } else { + nr_pdsch_symbol_processing(rdata); } - stop_meas(&gNB->dlsch_precoding_stats); + } + join_task_ans(&ans); + for (int i = 0; i < nb_tasks; i++) { + merge_meas(&gNB->dlsch_resource_mapping_stats, &arr[i].dlsch_resource_mapping_stats); + merge_meas(&gNB->dlsch_precoding_stats, &arr[i].dlsch_precoding_stats); } stop_meas(&gNB->dlsch_pdsch_generation_stats); /* output and its parts for each dlsch should be aligned on 64 bytes (or 8 * 64 bits) diff --git a/openair1/PHY/defs_gNB.h b/openair1/PHY/defs_gNB.h index b9d4992d32..a6fb9d8890 100644 --- a/openair1/PHY/defs_gNB.h +++ b/openair1/PHY/defs_gNB.h @@ -485,6 +485,7 @@ typedef struct PHY_VARS_gNB_s { notifiedFIFO_t L1_rx_out; tpool_t threadPool; int num_pusch_symbols_per_thread; + int num_pdsch_symbols_per_thread; int dmrs_num_antennas_per_thread; pthread_t L1_rx_thread; int L1_rx_thread_core; diff --git a/openair1/SIMULATION/NR_PHY/dlsim.c b/openair1/SIMULATION/NR_PHY/dlsim.c index c1fee991b9..2c08ab3b6e 100644 --- a/openair1/SIMULATION/NR_PHY/dlsim.c +++ b/openair1/SIMULATION/NR_PHY/dlsim.c @@ -382,6 +382,8 @@ int main(int argc, char **argv) uint8_t dlsch_threads = 0; int chest_type[2] = {0}; uint8_t max_ldpc_iterations = 5; + // number of PDSCH symbols per thread = 0 means do not use thread pool + int num_pdsch_symbols_per_thread = 0; if ((uniqCfg = load_configmodule(argc, argv, CONFIG_ENABLECMDLINEONLY)) == 0) { exit_fun("[NR_DLSIM] Error, configuration module init failed\n"); } @@ -583,6 +585,10 @@ int main(int argc, char **argv) gNBthreads[sizeof(gNBthreads)-1]=0; break; + case 'Y': + num_pdsch_symbols_per_thread = atoi(optarg); + break; + case 'Z' : filename_csv = strdup(optarg); AssertFatal(filename_csv != NULL, "strdup() error: errno %d\n", errno); @@ -657,6 +663,7 @@ int main(int argc, char **argv) printf("-T Enable PTRS, arguments list L_PTRS{0,1,2} K_PTRS{2,4}, e.g. -T 2 0 2 \n"); printf("-U Change DMRS Config, arguments list DMRS TYPE{0=A,1=B} DMRS AddPos{0:2} DMRS ConfType{1:2}, e.g. -U 3 0 2 1 \n"); printf("-X gNB thread pool configuration, n => no threads\n"); + printf("-Y Number of symbols processed per PDSCH generation thread\n"); printf("-Z Output filename (.csv format) for stats\n"); exit (-1); break; @@ -715,6 +722,7 @@ int main(int argc, char **argv) AssertFatal((gNB->if_inst = NR_IF_Module_init(0)) != NULL, "Cannot register interface"); gNB->if_inst->NR_PHY_config_req = nr_phy_config_request; + gNB->num_pdsch_symbols_per_thread = num_pdsch_symbols_per_thread; NR_ServingCellConfigCommon_t *scc = calloc(1,sizeof(*scc));; prepare_scc(scc); diff --git a/openair2/GNB_APP/L1_nr_paramdef.h b/openair2/GNB_APP/L1_nr_paramdef.h index 1faa03c475..9e33c57795 100644 --- a/openair2/GNB_APP/L1_nr_paramdef.h +++ b/openair2/GNB_APP/L1_nr_paramdef.h @@ -28,6 +28,10 @@ #define L1_MAX_LDPC_ITERATIONS "max_ldpc_iterations" #define L1_RX_THREAD_CORE "L1_rx_thread_core" #define L1_TX_THREAD_CORE "L1_tx_thread_core" +#define L1_NUM_RX_SYM_PER_THREAD "L1_num_rx_sym_per_thread" +#define HLP_L1_NUM_RX_SYM_PER_THREAD "number of symbols processed per PUSCH generation thread" +#define L1_NUM_TX_SYM_PER_THREAD "L1_num_tx_sym_per_thread" +#define HLP_L1_NUM_TX_SYM_PER_THREAD "number of symbols processed per PDSCH generation thread" #define HLP_TP_SIZ "thread_pool_size paramter removed, please use --thread-pool" #define L1_TX_AMP_BACKOFF_dB "tx_amp_backoff_dB" #define HLP_L1TX_BO "Backoff from full-scale output at the L1 entity(frequency domain), ex. 12 would corresponding to 14-bit input level (6 dB/bit). Default 36 dBFS for OAI RU entity" @@ -58,6 +62,8 @@ {L1_MAX_LDPC_ITERATIONS, NULL, 0, .uptr=NULL, .defintval=8, TYPE_UINT, 0}, \ {L1_RX_THREAD_CORE, NULL, 0, .iptr=NULL, .defintval=-1, TYPE_INT, 0}, \ {L1_TX_THREAD_CORE, NULL, 0, .iptr=NULL, .defintval=-1, TYPE_INT, 0}, \ + {L1_NUM_RX_SYM_PER_THREAD, HLP_L1_NUM_RX_SYM_PER_THREAD, 0, .iptr=NULL, .defintval=1, TYPE_INT, 0}, \ + {L1_NUM_TX_SYM_PER_THREAD, HLP_L1_NUM_TX_SYM_PER_THREAD, 0, .iptr=NULL, .defintval=0, TYPE_INT, 0}, \ {L1_TX_AMP_BACKOFF_dB, HLP_L1TX_BO,0, .uptr=NULL, .defintval=36, TYPE_UINT, 0}, \ {L1_PHASE_COMP, HLP_L1_PHASE_COMP,PARAMFLAG_BOOL, .uptr=NULL,.defintval=1, TYPE_UINT, 0}, \ {L1_NUM_ANTENNAS_PER_THREAD, HLP_NUM_ARX,0, .uptr=NULL, .defintval=1, TYPE_UINT, 0}, \ diff --git a/openair2/GNB_APP/gnb_config.c b/openair2/GNB_APP/gnb_config.c index 4928d98183..a73a77b0ee 100644 --- a/openair2/GNB_APP/gnb_config.c +++ b/openair2/GNB_APP/gnb_config.c @@ -841,6 +841,9 @@ void RCconfig_NR_L1(void) gNB->L1_rx_thread_core = *gpd(params, np, L1_RX_THREAD_CORE)->iptr; gNB->L1_tx_thread_core = *gpd(params, np, L1_TX_THREAD_CORE)->iptr; LOG_I(NR_PHY, "thread cores for L1_RX %d L1_TX %d\n", gNB->L1_rx_thread_core, gNB->L1_tx_thread_core); + // PUSCH symbols per thread need to be calculated by how many threads we have + gNB->num_pusch_symbols_per_thread = *gpd(params, np, L1_NUM_RX_SYM_PER_THREAD)->iptr; + gNB->num_pdsch_symbols_per_thread = *gpd(params, np, L1_NUM_TX_SYM_PER_THREAD)->iptr; gNB->TX_AMP = min(32767.0 / pow(10.0, .05 * (double)(*gpd(params, np, L1_TX_AMP_BACKOFF_dB)->uptr)), INT16_MAX); LOG_I(NR_PHY, "TX_AMP = %d (-%d dBFS)\n", gNB->TX_AMP, *gpd(params, np, L1_TX_AMP_BACKOFF_dB)->uptr); AssertFatal(gNB->TX_AMP > 300, "TX_AMP is too small, must be larger than 300 (is %d)\n", gNB->TX_AMP);