maki/openairinterface5g
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ldpc_t2: generalize for broader BBDEV support (ACC100/200)

Browse Source 
- Generalized LDPC T2 implementation to support other DPDK BBDEV
  accelerators (e.g., Intel ACC100/200)
- Added BBDEV capability checks during initialization
- Introduced LLR scaling to improve decoding performance on Intel ACC
  cards
- Implemented self-managed HARQ buffers for ACC200 (no internal HARQ
  memory)
- Library usage documentation update, and added developer documentation

Co-authored-by: Romain Beurdouche <romain.beurdouche@eurecom.fr>
This commit is contained in:
Xin Zhe Khooi
2025-08-05 18:37:58 +02:00
committed by Robert Schmidt
co-authored by Romain Beurdouche
parent b763692792
commit 61ea0a9e81
6 changed files with 928 additions and 239 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LDPC_T2_OFFLOAD_SETUP.md
+244 -63
View File
@@ -7,7 +7,7 @@
</a>
</td>
<td style="border-collapse: collapse; border: none; vertical-align: center;">
<b><font size = "5">OAI T1/T2 LDPC offload</font></b>
<b><font size = "5">OAI LDPC offload (O-RAN AAL/DPDK BBDEV)</font></b>
</td>
</tr>
</table>
@@ -16,19 +16,57 @@
[[_TOC_]]
This documentation aims to provide a tutorial for AMD Xilinx T2 Telco card integration into OAI and its usage.
This documentation describes the integration of LDPC coding for lookaside acceleration using O-RAN AAL/DPDK BBDEV in OAI, along with its usage.
For details on the implementation, please consult the [developer notes](../openair1/PHY/CODING/nrLDPC_coding/nrLDPC_coding_t2/README.md).
# Requirements
- bitstream image and PMD driver for the T2 card provided by AccelerComm
- DPDK 20.11.9 with patch from Accelercomm: version ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch
- tested on RHEL7.9, RHEL9.2, Ubuntu 22.04
In principle, any lookaside LDPC accelerator supporting the O-RAN AAL/DPDK BBDEV should work.
However, the current implementation has only been validated for the Xilinx T2, Intel ACC100, and Intel ACC200 (VRB1).
Therefore, your mileage may vary when using other BBDEV devices as there may be some hardware-specific changes required -- contributions are welcome!
# DPDK setup
## DPDK Version Requirements
The following DPDK versions are supported:
- For the Xilinx T2 card, DPDK20.11+ is supported.
- As for the Intel ACC100/ACC200, only DPDK22.11+ is supported.
## Tested Devices/ DPDK versions
### Xilinx T2
- DPDK20.11.9*.
- DPDK22.11.7*.
> Note: FPGA bitstream image and the corresponding patch file (e.g., `ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch` for DPDK20.11) from Accelercomm required.
### Intel ACC100
- DPDK22.11.7*.
- DPDK23.11.3*.
- DPDK24.11.2.
> Note: [Patch]((https://github.com/DPDK/dpdk/commit/fdde63a1dfc129d0a510a831aa98253b36a2a1cd)) required for pre-DPDK24.11 versions when using the Intel ACC100.
### Intel ACC200 (also known as VRB1)
- DPDK22.11.7.
- DPDK23.11.3.
- DPDK24.11.2.
# System Setup
## DPDK installation
*Note: Following instructions are valid for ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch version, which is compatible with DPDK 20.11.9. Installation steps, which should be followed for older versions of the patch file (for example ACL_BBDEV_DPDK20.11.3_BL_1006_build_1105_dev_branch_MCT_optimisations_1106_physical_std.patch) are present in older version of this documentation, under the tag 2023.w48.*
```
> Important:
> - If you are using the Xilinx T2 card, you will need to apply the vendor-supplied patches before compiling DPDK.
> - If you are using the Intel ACC100, you will need to [patch](https://github.com/DPDK/dpdk/commit/fdde63a1dfc129d0a510a831aa98253b36a2a1cd) the ACC100's driver if you are using DPDK22.11 or DPDK23.11.
Refer to the guide [here](./ORAN_FHI7.2_Tutorial.md?ref_type=heads#dpdk-data-plane-development-kit) to install, and then validate your DPDK installation.
<details open>
<summary> Notes on DPDK patching/installation for Xilinx T2. </summary>
*Note: The following instructions apply to `ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch`, compatible with DPDK 20.11.9. For older patches (e.g., `ACL_BBDEV_DPDK20.11.3_BL_1006_build_1105_dev_branch_MCT_optimisations_1106_physical_std.patch`), refer to the T2 documentation in `2023.w48`.*
```bash
# Get DPDK source code
git clone https://github.com/DPDK/dpdk-stable.git ~/dpdk-stable
cd ~/dpdk-stable
@@ -37,50 +75,127 @@ git apply ~/ACL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch
```
Replace `~/ACL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch` by patch file provided by
Accelercomm.
```
If you would like to install DPDK to a custom directory, here is an example.
```bash
cd ~/dpdk-stable
meson setup build
# meson setup --prefix=/opt/dpdk-t2 build for installation with non-default installation prefix
# meson setup build
meson setup --prefix=/opt/dpdk-t2 build # for installation with non-default installation prefix
cd build
ninja
sudo ninja install
sudo ldconfig
```
## DPDK configuration
- load required kernel module
</details>
## System configuration
### Setting up Hugepages
First, we must setup hugepages on the system.
In our setup, we setup 16 of the 1G hugepages.
Apart from 1G, 2MB hugepages works too, but make sure to allocate a sufficient number of them.
```
sudo modprobe vfio-pci
```
- check presence of the card and its PCI addres on the host machine
```
lspci | grep "Xilinx"
```
- bind the card with vfio-pci driver
```
sudo python3 ~/dpdk-stable/usertools/dpdk-devbind.py --bind=vfio-pci 41:00.0
```
Replace PCI address of the card *41:00.0* by address detected by *lspci | grep "Xilinx"* command
- hugepages setup (10 x 1GB hugepages)
```
sudo python3 ~/dpdk-stable/usertools/dpdk-hugepages.py -p 1G --setup 10G
# sudo dpdk-hugepages.py -p 1G --setup 16G
```
*Note: device binding and hugepages setup has to be done after every reboot of
the host machine*
### Locating the Accelerator
# Modifications in the OAI code
## DPDK lib and PMD path specification
If DPDK library was installed into custom path, you have to point to the right directory with `PKG_CONFIG_PATH`, prior to the OAI build. Sample command to set the DPDK path to `/opt/dpdk-t2/lib64/pkgconfig/`:
Next, we check whether our system can detect our accelerator using `dpdk-devbind.py`.
You should see Baseband devices detected by DPDK, as follows:
```
export PKG_CONFIG_PATH=/opt/dpdk-t2/lib64/pkgconfig/:$PKG_CONFIG_PATH
# sudo dpdk-devbind.py -s
...
Baseband devices using DPDK-compatible driver
=============================================
0000:f7:00.0 'Device 57c0' unused=vfio-pci
...
```
# OAI Build
As you can see here, our Intel ACC200 has the address of `0000:f7:00.0`.
Depending on the accelerator you are using, the address may vary.
### Loading VFIO-PCI and enabling SR-IOV
Following, make sure to load the `vfio-pci` kernel modules and ensure that SR-IOV is enabled.
```
# sudo modprobe vfio-pci enable_sriov=1 disable_idle_d3=1
```
### Binding the Accelerator with `vfio-pci`
Lastly, we bind our accelerator with the `vfio-pci` driver.
```
# sudo dpdk-devbind.py --bind=vfio-pci 0000:f7:00.0
```
> Note: For the Xilinx T2, we can use this device directly.
If you use an Intel vRAN accelerator, read on.
### Additional Steps for Intel vRAN Accelerators
> IMPORTANT NOTE:
> - Currently, we only support using the Virtual Functions (VFs) of the Intel vRAN accelerators, but not the Physical Function (PF).
> - One key advantage of using VFs is that this allows us to share the accelerator with other DU instances on the same machine, which is common in practice.
If you are using an Intel vRAN accelerator, you will need to use the [pf_bb_config](https://github.com/intel/pf-bb-config) tool to configure the accelerator beforehand.
#### pf_bb_config
For more details, please consult the `pf_bb_config` README.
```
# git clone https://github.com/intel/pf-bb-config
# cd ~/pf-bb-config
# ./build.sh
```
This clones and builds the `pf_bb_config` binary.
Next, we show an example for the Intel ACC200.
We use an existing configuration located at `./vrb1/vrb1_config_16vf.cfg`.
Here, it is necessary to specify a VFIO token (in this case, we use the UUID `00112233-4455-6677-8899-aabbccddeeff`).
Note that in practice, a random UUID should be used.
```
# sudo ./pf_bb_config VRB1 -v 00112233-4455-6677-8899-aabbccddeeff -c vrb1/vrb1_config_16vf.cfg
== pf_bb_config Version v25.01-0-g812e032 ==
VRB1 PF [0000:f7:00.0] configuration complete!
Log file = /var/log/pf_bb_cfg_0000:f7:00.0.log
```
#### Creating VFs
Finally, we create the VF(s) for our accelerator.
In this example, we only create one SR-IOV VF.
```
# echo 1 | sudo tee /sys/bus/pci/devices/0000:f7:00.0/sriov_numvfs
```
If you encounter any error when creating the VF(s), e.g., `tee: '/sys/bus/pci/devices/0000:f7:00.0/sriov_numvfs': No such file or directory`, then try enabling SR-IOV again.
```
# echo 1 | sudo tee /sys/module/vfio_pci/parameters/enable_sriov
```
After you have successfully created the VF, you should see an additional baseband device, in our case, it is `0000:f7:00.1`.
We will use this device with OAI later.
The newly created VF should also be using the same `vfio-pci` driver as the PF, if it is not, you will need to do a `dpdk-devbind.py` to bind it with `vfio-pci`.
```
# sudo dpdk-devbind.py -s
...
Baseband devices using DPDK-compatible driver
=============================================
0000:f7:00.0 'Device 57c0' drv=vfio-pci unused=
0000:f7:00.1 'Device 57c0' drv=vfio-pci unused=
...
```
# Building OAI with ORAN-AAL
OTA deployment is precisely described in the following tutorial:
- [NR_SA_Tutorial_COTS_UE](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/doc/NR_SA_Tutorial_COTS_UE.md)
Instead of section *3.2 Build OAI gNB* from the tutorial, run following commands:
Instead of section *3.2 Build OAI gNB* from the tutorial, run the following commands:
```
```bash
# Get openairinterface5g source code
git clone https://gitlab.eurecom.fr/oai/openairinterface5g.git ~/openairinterface5g
cd ~/openairinterface5g
@@ -97,21 +212,35 @@ cd cmake_targets
./build_oai -w USRP --ninja --gNB -P --build-lib "ldpc_t2" -C
```
Shared object file *libldpc_t2.so* is created during the compilation. This object is conditionally compiled. Selection of the library to compile is done using *--build-lib ldpc_t2*.
A shared object file `libldpc_t2.so` will be created during the compilation.
This object is conditionally compiled.
The selection of the library to compile is done using `--build-lib ldpc_t2`.
*Required poll mode driver has to be present on the host machine and required DPDK version has to be installed on the host, prior to the build of OAI*
> Note: The required DPDK poll mode driver has to be present on the host machine and required DPDK version has to be installed on the host, prior to building OAI.
# Setup of T2-related DPDK EAL parameters
To configure T2-related DPDK Environment Abstraction Layer (EAL) parameters, you can set the following parameters via the command line of PHY simulators or softmodem:
- `nrLDPC_coding_t2.dpdk_dev` - **mandatory** parameter, specifies PCI address of the T2 card. PCI address of the T2 card can be detected by `lspci | grep "Xilinx"` command.
- `nrLDPC_coding_t2.dpdk_core_list` - **mandatory** parameter, specifies CPU cores assigned to DPDK for T2 processing. Ensure that the CPU cores specified in *nrLDPC_coding_t2.dpdk_core_list* are available and not used by other processes to avoid conflicts.
- `nrLDPC_coding_t2.dpdk_prefix` - DPDK shared data file prefix, by default set to *b6*.
# O-RAN AAL DPDK EAL parameters
To configure O-RAN AAL/DPDK BBDEV, you can set the following parameters via the command line of PHY simulators or nr-softmodem:
**Note:** These parameters can also be provided in a configuration file:
- `nrLDPC_coding_t2.dpdk_dev` - **mandatory** parameter, specifies the PCI address of our accelerator. It must follow the format `WWWW:XX:YY.Z`.
- `nrLDPC_coding_t2.dpdk_core_list` - **mandatory** parameter, specifies the CPU cores assigned to DPDK .
Ensure that the CPU cores specified in `nrLDPC_coding_t2.dpdk_core_list` are available and not used by other processes to avoid conflicts.
- `nrLDPC_coding_t2.dpdk_prefix` - optional parameter, DPDK shared data file prefix, by default set to *b6*.
- `nrLDPC_coding_t2.vfio_vf_token` - optional parameter, VFIO token set for the VF, if applicable.
- `nrLDPC_coding_t2.num_harq_codeblock` - optional parameter, size of the HARQ buffer in terms of the number of 32kB blocks, by default set to *512* (maximum for the T2; as for the ACCs, this can be further increased).
- `nrLDPC_coding_t2.eal_init_bbdev` - optional parameter, set this to 1 when using with FHI72 with the Intel ACCs. For the T2, this is not required.
**Note:** These parameters can also be provided in a configuration file.
Example for the ACC200:
```
nrLDPC_coding_t2 : {
dpdk_dev : "41:00.0";
dpdk_dev : "0000:f7:00.1";
dpdk_core_list : "14-15";
vfio_vf_token: "00112233-4455-6677-8899-aabbccddeeff";
};
loader : {
@@ -121,38 +250,90 @@ loader : {
};
```
# 5G PHY simulators
# Running OAI with O-RAN AAL
## nr_ulsim test
Offload of the channel decoding to the T2 card is in nr_ulsim specified by *--loader.ldpc.shlibversion _t2* option. Example command for running nr_ulsim with LDPC decoding offload to the T2 card:
```
In general, to offload of the channel coding to the LDPC accelerator, we use use the `--loader.ldpc.shlibversion _t2` option.
## 5G PHY simulators
### nr_ulsim
Example command:
```bash
cd ~/openairinterface5g
source oaienv
cd cmake_targets/ran_build/build
sudo ./nr_ulsim -n100 -s20 -m20 -r273 -R273 --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 01:00.0 --nrLDPC_coding_t2.dpdk_core_list 0-1
```
## nr_dlsim test
Offload of the channel encoding to the AMD Xilinx T2 card is in nr_dlsim specified by *-c* option. Example command for running nr_dlsim with LDPC encoding offload to the T2 card:
sudo ./nr_ulsim -n100 -s20 -m20 -r273 -R273 --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 0000:f7:00.1 --nrLDPC_coding_t2.dpdk_core_list 0-1 --nrLDPC_coding_t2.vfio_vf_token 00112233-4455-6677-8899-aabbccddeeff
```
### nr_dlsim
Example command:
```bash
cd ~/openairinterface5g
source oaienv
cd cmake_targets/ran_build/build
sudo ./nr_dlsim -n300 -s30 -R 106 -e 27 --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 01:00.0 --nrLDPC_coding_t2.dpdk_core_list 0-1
sudo ./nr_dlsim -n300 -s30 -R 106 -e 27 --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 0000:f7:00.1 --nrLDPC_coding_t2.dpdk_core_list 0-1 --nrLDPC_coding_t2.vfio_vf_token 00112233-4455-6677-8899-aabbccddeeff
```
# OTA test
Offload of the channel encoding and decoding to the AMD Xilinx T2 card is enabled by *--loader.ldpc.shlibversion _t2* option.
## OTA test
## Run OAI gNB with USRP B210
```
### Running OAI gNB with USRP B210/FHI72
Example command:
```bash
cd ~/openairinterface5g
source oaienv
cd cmake_targets/ran_build/build
sudo ./nr-softmodem -O ../../../targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb.sa.band78.fr1.106PRB.usrpb210.conf --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 01:00.0 --nrLDPC_coding_t2.dpdk_core_list 0-1
sudo ./nr-softmodem -O ~/gnb.conf --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 0000:f7:00.1 --nrLDPC_coding_t2.dpdk_core_list 14-15 --nrLDPC_coding_t2.vfio_vf_token 00112233-4455-6677-8899-aabbccddeeff --nrLDPC_coding_t2.eal_init_bbdev 1
```
# Limitations
## AMD Xilinx T2 card
- functionality of the LDPC encoding and decoding offload verified in OTA SISO setup with USRP N310 and Quectel RM500Q, blocking of the card reported for MIMO setup (2 layers)
# Known Issue(s)
*Note: AMD Xilinx T1 Telco card is not supported anymore.*
## BBDEV CPU Usage
When running the E2E setup (this applies to both USRP and FHI72), BBDEV may not be using the list of CPU cores as specified by `nrLDPC_coding_t2.dpdk_core_list` accordingly.
This is an issue under investigation, and subject for future fixes.
In the meantime, we recommend allocating idle, and isolated CPU cores in the configuration for BBDEV.
## Potential Low Throughput
The current implementation has been tested to work in an end-to-end setup and is functional.
However, there are still opportunities for optimization, particularly in LDPC decoding performance, which is an area of ongoing improvement.
As such, downlink/uplink throughput may be suboptimal with the default configurations, but enhancements are actively being explored.
To achieve better E2E performance with the current implementation, we recommend the following adjustments:
1. Increasing the number of LDPC decoding iterations of the L1, e.g., `max_ldpc_iterations` to 200.
2. Increasing the BLER targets of the MAC scheduler.
Example configuration snippet:
```
...
MACRLCs = (
{
num_cc = 1;
tr_s_preference = "local_L1";
tr_n_preference = "local_RRC";
pusch_TargetSNRx10 = 180;
pucch_TargetSNRx10 = 220;
dl_bler_target_upper = .35;
dl_bler_target_lower = .15;
ul_bler_target_upper = .35;
ul_bler_target_lower = .15;
pusch_FailureThres = 1000;
ul_max_mcs = 28;
}
);
L1s = (
{
num_cc = 1;
tr_n_preference = "local_mac";
prach_dtx_threshold = 120;
pucch0_dtx_threshold = 100;
ofdm_offset_divisor = 8; #set this to UINT_MAX for offset 0
max_ldpc_iterations = 200;
}
);
...
```