From 7d7dced70e97f142041d046f39324530f0f3f38f Mon Sep 17 00:00:00 2001 From: Jaroslava Fiedlerova Date: Wed, 29 Nov 2023 09:12:30 +0100 Subject: [PATCH] T2 offload - documentation update --- doc/LDPC_T2_OFFLOAD_SETUP.md | 159 ++++++++++++++++++++++++----------- 1 file changed, 112 insertions(+), 47 deletions(-) diff --git a/doc/LDPC_T2_OFFLOAD_SETUP.md b/doc/LDPC_T2_OFFLOAD_SETUP.md index c8fdbf447a..551ddcc1df 100644 --- a/doc/LDPC_T2_OFFLOAD_SETUP.md +++ b/doc/LDPC_T2_OFFLOAD_SETUP.md @@ -17,68 +17,133 @@ [[_TOC_]] This documentation aims to provide a tutorial for AMD Xilinx T2 Telco card integration into OAI and its usage. -## Prerequisites -Offload of the channel decoding was tested with the T2 card in following setup: -**AMD Xilinx T2 card** - - bitstream image and PMD driver provided by AccelerComm - - DPDK 20.11.3 with patch from Accelercomm (also tested with DPDK 20.11.7 - version) +# Requirements + + - bitstream image and PMD driver for the T2 card provided by AccelerComm + - DPDK 20.11.7 with patch from Accelercomm (also tested with DPDK 20.11.3) - tested on RHEL7.9, RHEL9.2, Ubuntu 20.04, Ubuntu 22.04 -## DPDK setup - - check the presence of the card on the host - - `lspci | grep "Xilinx"` - - binding of the device with igb_uio driver - - `./dpdk-devbind.py -b igb_uio
` +# DPDK setup +## DPDK installation +``` +# Get DPDK source code +git clone https://github.com/DPDK/dpdk-stable.git ~/dpdk-stable +cd ~/dpdk-stable +git checkout v20.11.7 +git apply ~/ACL_BBDEV_DPDK20.11.3_xxx.patch +``` +Replace `~/ACL_BBDEV_DPDK20.11.3_xxx.patch` by patch file provided by +Accelercomm. +``` +cd ~/dpdk-stable +meson setup build +# meson setup --prefix=/opt/dpdk-t2 build for installation with non-default installation prefix +cd build +ninja +sudo ninja install +sudo ldconfig +``` +## DPDK configuration + - load required kernel module +``` +sudo modprobe igb_uio +sudo insmod ~/dpdk-stable/kernel/linux/igb_uio/igb_uio.ko +``` + - check presence of the card and its PCI addres on the host machine +``` +lspci | grep "Xilinx" +``` + - bind the card with igb_uio driver +``` +sudo python3 ~/dpdk-stable/usertools/dpdk-devbind.py -b igb_uio 41:00.0 +``` +Replace PCI address of the card *41:00.0* by address detected by *lspci | grep "Xilinx"* command - hugepages setup (10 x 1GB hugepages) - - `./dpdk-hugepages.py -p 1G --setup 10G` +``` +sudo python3 ~/dpdk-stable/usertools/dpdk-hugepages.py -p 1G --setup 10G` +``` -*Note: Commands to run from dpdk/usertool folder* +*Note: device binding and hugepages setup has to be done after every reboot of +the host machine* -## Compilation -Deployment of the OAI is precisely described in the following tutorials. - - [BUILD](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/doc/BUILD.md) - - [NR_SA_CN5G_gNB_USRP_COTS_UE_Tutorial](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/doc/NR_SA_Tutorial_COTS_UE.md) +# Modifications in the OAI code +## PMD path specification +Path to the PMD for operating the card is specified in `CMakeLists.txt` file in +*LDPC OFFLOAD library* section. Modify following line based on the location of +the PMD on your system. By deafult, path to the PMD is set to `/opt/dpdk-t2/lib/x86_64-linux-gnu/`. +``` +find_library(PMD_T2 NAMES rte_baseband_accl_ldpc HINTS "/opt/dpdk-t2/lib/x86_64-linux-gnu/") +``` -Shared object file *libldpc_t2.so* is created during the compilation. This object is conditionally compiled. Selection of the library to compile is done using *--build-lib ldpc_t2*. Example command to build OAI with support for LDPC offload to T2 card is: +## T2 card DPDK initialization +Following lines in `openair1/PHY/CODING/nrLDPC_decoder/nrLDPC_decoder_offload.c` file has to be +modified based on your system requirements. By default, PCI address of the T2 card is set to 41:00.0 and cores 14 and 15 are assigned to the DPDK. +``` + char *dpdk_dev = "41:00.0"; //PCI address of the card + char *argv_re[] = {"bbdev", "-a", dpdk_dev, "-l", "14-15", "--file-prefix=b6", "--"}; +``` +For the DPDK EAL initialization, device is specified by `-a` option and list +of cores to run the DPDK application on is selected by `-l` option. PCI adress of +the T2 card can be detected by `lspci | grep "Xilinx"` command. -`./build_oai -P --gNB -w USRP --build-lib "ldpc_t2" --ninja` +# OAI Build +OTA deployment is precisely described in the following tutorial: +- [NR_SA_Tutorial_COTS_UE](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/doc/NR_SA_Tutorial_COTS_UE.md) +Instead of section *3.2 Build OAI gNB* from the tutorial, run following commands: + +``` +# Get openairinterface5g source code +git clone https://gitlab.eurecom.fr/oai/openairinterface5g.git ~/openairinterface5g +cd ~/openairinterface5g +git checkout develop + +# Install OAI dependencies +cd ~/openairinterface5g/cmake_targets +./build_oai -I + +# Build OAI gNB +cd ~/openairinterface5g +source oaienv +cd cmake_targets +./build_oai -w USRP --ninja --gNB -P -C --build-lib "ldpc_t2" +``` +Shared object file *libldpc_t2.so* is created during the compilation. This object is conditionally compiled. Selection of the library to compile is done using *--build-lib ldpc_t2*. *Required poll mode driver has to be present on the host machine and required DPDK version has to be installed on the host, prior to the build of OAI* +# 5G PHY simulators ## nr_ulsim test -Offload of the channel decoding to the AMD Xilinx T2 card is in nr_ulsim specified by *-o* option. Example command for running nr_ulsim with LDPC decoding offload to the T2 card: - -`sudo ./nr_ulsim -n100 -s20 -m20 -r273 -o` - +Offload of the channel decoding to the T2 card is in nr_ulsim specified by *-o* option. Example command for running nr_ulsim with LDPC decoding offload to the T2 card: +``` +cd ~/openairinterface5g +source oaienv +cd cmake_targets/ran_build/build +sudo ./nr_ulsim -n100 -s20 -m20 -r273 -R273 -o +``` ## nr_dlsim test Offload of the channel encoding to the AMD Xilinx T2 card is in nr_dlsim specified by *-c* option. Example command for running nr_dlsim with LDPC encoding offload to the T2 card: +``` +cd ~/openairinterface5g +source oaienv +cd cmake_targets/ran_build/build +sudo ./nr_dlsim -n300 -s30 -R 106 -e 27 -c +``` -`sudo ./nr_dlsim -n300 -s30 -R 106 -e 27 -c` +# OTA test +Offload of the channel encoding and decoding to the AMD Xilinx T2 card is enabled by *--ldpc-offload-enable 1* option. -## OTA and RFSIM test -Offload of the channel encoding and decoding to the AMD Xilinx T2 card is enabled by *--ldpc-offload-enable 1* option. Example command for running nr-softmodem with LDPC processing on the T2 card: +## Run OAI gNB with USRP B210 +``` +cd ~/openairinterface5g +source oaienv +cd cmake_targets/ran_build/build +sudo ./nr-softmodem --sa -O ../../../targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb.sa.band78.fr1.106PRB.usrpb210.conf --ldpc-offload-enable 1 +``` -`./nr-softmodem -O config_file.conf --sa --ldpc-offload-enable 1` +# Limitations +## AMD Xilinx T2 card + - offload of the LDPC encoding implemented for MCS > 2, OAI CPU encoder is used for MCS =< 2 + - functionality of the LDPC encoding and decoding offload verified in OTA SISO setup with USRP N310 and Quectel RM500Q, blocking of the card reported for MIMO setup (2 layers) -### LDPC encoding/decoding with CPU/GPU -Available LDPC implementations and loading particular libraries is well described in *openair1/PHY/CODING/DOC/LDPCImplementation.md*. Default library for LDPC processing is called `libldpc.so`. Sample command for running nr-softmodem with selected library (in this case we want to replace `libldpc.so` by `libldpc_optim.so` library): -`./nr-softmodem -O config_file.conf --sa --loader.ldpc.shlibversion _optim` - -Available options: -- --loader.ldpc.shlibversion _optim8seg -- --loader.ldpc.shlibversion _optim -- --loader.ldpc.shlibversion _orig -- --loader.ldpc.shlibversion _cuda -- --loader.ldpc.shlibversion _cl - -## Limitations -### AMD Xilinx T1 card - - offload of the LDPC decoding implemented only for MCS > 9, decoding of the smaller TBs on the T1 card leads to blocking of the card - - HARQ is not functional with LDPC decoding offload to the T1 card (restricted in the code to avoid blocking of the card) - - LDPC encoding offload not implemented for the T1 card - - not supported in current develop branch, please checkout to *2023.w43* -### AMD Xilinx T2 card - - offload of the LDPC encoding implemented for MCS > 2 - - occasional blocking of the card in decoder function - issue is investigated +*Note: AMD Xilinx T1 Telco card is not supported anymore.*