maki/openairinterface5g
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/ldpc-aal' into integration_2025_w32 (!3344)

Browse Source 
LDPC BBDEV/AAL rework

This branch aims to generalize the existing LDPC T2 implementation to support
other DPDK BBDEV accelerators, e.g., Intel ACC100/200.

Differences from the current T2 implementation include:
  1. relax CMakeLists.txt checking for custom dpdk-t2 implementation,
     technically, any DPDK BBDEV should work
  2. BBDEV capability checks during init
  3. llr_scaling to scale the input llr accordingly for better decoding
     performance on the Intel ACC100/200
  4. self-managed harq buffers for the ACC200, as internal harq memory is not
     available

Tested BBDEV devices:
  1. Intel ACC100
  2. Intel ACC200/VRB1
This commit is contained in:
Jaroslava Fiedlerova
2025-08-07 12:17:11 +02:00
parent 5555a465fc f6aeb21d97
commit 5ca59d0f48
16 changed files with 1097 additions and 415 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/BUILD.md
+1 -1
View File
@@ -185,7 +185,7 @@ Some libraries have further dependencies and might not build on every system:
- `enbscope`, `uescope`, `nrscope`: libforms/X
- `ldpc_cuda`: CUDA
- `websrv`: npm and others
- `ldpc_t2`: DPDK with patch
- `ldpc_aal`: DPDK with patch
# Running `cmake` directly
doc/LDPC_OFFLOAD_SETUP.md
+344
View File
@@ -0,0 +1,344 @@
<table style="border-collapse: collapse; border: none;">
<tr style="border-collapse: collapse; border: none;">
<td style="border-collapse: collapse; border: none;">
<a href="http://www.openairinterface.org/">
<img src="./images/oai_final_logo.png" alt="" border=3 height=50 width=150>
</img>
</a>
</td>
<td style="border-collapse: collapse; border: none; vertical-align: center;">
<b><font size = "5">OAI LDPC offload (O-RAN AAL/DPDK BBDEV)</font></b>
</td>
</tr>
</table>
**Table of Contents**
[[_TOC_]]
This documentation describes the integration of LDPC coding for lookaside acceleration using O-RAN AAL/DPDK BBDEV in OAI, along with its usage.
For details on the implementation, please consult the [developer notes](../openair1/PHY/CODING/nrLDPC_coding/nrLDPC_coding_aal/README.md).
# Requirements
In principle, any lookaside LDPC accelerator supporting the O-RAN AAL/DPDK BBDEV should work.
However, the current implementation has only been validated for the Xilinx T2, Intel ACC100, and Intel ACC200 (VRB1).
Therefore, your mileage may vary when using other BBDEV devices as there may be some hardware-specific changes required -- contributions are welcome!
## DPDK Version Requirements
The following DPDK versions are supported:
- For the Xilinx T2 card, DPDK20.11+ is supported.
- As for the Intel ACC100/ACC200, only DPDK22.11+ is supported.
## Tested Devices/ DPDK versions
### Xilinx T2
- DPDK20.11.9*.
- DPDK22.11.7*.
> Note: FPGA bitstream image and the corresponding patch file (e.g., `ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch` for DPDK20.11) from Accelercomm required.
### Intel ACC100
- DPDK22.11.7*.
- DPDK23.11.3*.
- DPDK24.11.2.
> Note: [Patch]((https://github.com/DPDK/dpdk/commit/fdde63a1dfc129d0a510a831aa98253b36a2a1cd)) required for pre-DPDK24.11 versions when using the Intel ACC100.
### Intel ACC200 (also known as VRB1)
- DPDK22.11.7.
- DPDK23.11.3.
- DPDK24.11.2.
# System Setup
## DPDK installation
> Important:
> - If you are using the Xilinx T2 card, you will need to apply the vendor-supplied patches before compiling DPDK.
> - If you are using the Intel ACC100, you will need to [patch](https://github.com/DPDK/dpdk/commit/fdde63a1dfc129d0a510a831aa98253b36a2a1cd) the ACC100's driver if you are using DPDK22.11 or DPDK23.11.
Refer to the guide [here](./ORAN_FHI7.2_Tutorial.md?ref_type=heads#dpdk-data-plane-development-kit) to install, and then validate your DPDK installation.
<details open>
<summary> Notes on DPDK patching/installation for Xilinx T2. </summary>
*Note: The following instructions apply to `ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch`, compatible with DPDK 20.11.9. For older patches (e.g., `ACL_BBDEV_DPDK20.11.3_BL_1006_build_1105_dev_branch_MCT_optimisations_1106_physical_std.patch`), refer to the T2 documentation in `2023.w48`.*
```bash
# Get DPDK source code
git clone https://github.com/DPDK/dpdk-stable.git ~/dpdk-stable
cd ~/dpdk-stable
git checkout v20.11.9
git apply ~/ACL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch
```
Replace `~/ACL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch` by patch file provided by
Accelercomm.
If you would like to install DPDK to a custom directory, here is an example.
```bash
cd ~/dpdk-stable
# meson setup build
meson setup --prefix=/opt/dpdk-t2 build # for installation with non-default installation prefix
cd build
ninja
sudo ninja install
sudo ldconfig
```
</details>
## System configuration
### Setting up Hugepages
First, we must setup hugepages on the system.
In our setup, we setup 16 of the 1G hugepages.
Apart from 1G, 2MB hugepages works too, but make sure to allocate a sufficient number of them.
```
# sudo dpdk-hugepages.py -p 1G --setup 16G
```
### Locating the Accelerator
Next, we check whether our system can detect our accelerator using `dpdk-devbind.py`.
You should see Baseband devices detected by DPDK, as follows:
```
# sudo dpdk-devbind.py -s
...
Baseband devices using DPDK-compatible driver
=============================================
0000:f7:00.0 'Device 57c0' unused=vfio-pci
...
```
As you can see here, our Intel ACC200 has the address of `0000:f7:00.0`.
Depending on the accelerator you are using, the address may vary.
### Loading VFIO-PCI and enabling SR-IOV
Following, make sure to load the `vfio-pci` kernel modules and ensure that SR-IOV is enabled.
```
# sudo modprobe vfio-pci enable_sriov=1 disable_idle_d3=1
```
### Binding the Accelerator with `vfio-pci`
Lastly, we bind our accelerator with the `vfio-pci` driver.
```
# sudo dpdk-devbind.py --bind=vfio-pci 0000:f7:00.0
```
> Note: For the Xilinx T2, we can use this device directly.
If you use an Intel vRAN accelerator, read on.
### Additional Steps for Intel vRAN Accelerators
> IMPORTANT NOTE:
> - Currently, we only support using the Virtual Functions (VFs) of the Intel vRAN accelerators, but not the Physical Function (PF).
> - One key advantage of using VFs is that this allows us to share the accelerator with other DU instances on the same machine, which is common in practice.
If you are using an Intel vRAN accelerator, you will need to use the [pf_bb_config](https://github.com/intel/pf-bb-config) tool to configure the accelerator beforehand.
#### pf_bb_config
For more details, please consult the `pf_bb_config` README.
```
# git clone https://github.com/intel/pf-bb-config
# cd ~/pf-bb-config
# ./build.sh
```
This clones and builds the `pf_bb_config` binary.
Next, we show an example for the Intel ACC200.
We use an existing configuration located at `./vrb1/vrb1_config_16vf.cfg`.
Here, it is necessary to specify a VFIO token (in this case, we use the UUID `00112233-4455-6677-8899-aabbccddeeff`).
Note that in practice, a random UUID should be used.
```
# sudo ./pf_bb_config VRB1 -v 00112233-4455-6677-8899-aabbccddeeff -c vrb1/vrb1_config_16vf.cfg
== pf_bb_config Version v25.01-0-g812e032 ==
VRB1 PF [0000:f7:00.0] configuration complete!
Log file = /var/log/pf_bb_cfg_0000:f7:00.0.log
```
#### Creating VFs
Finally, we create the VF(s) for our accelerator.
In this example, we only create one SR-IOV VF.
```
# echo 1 | sudo tee /sys/bus/pci/devices/0000:f7:00.0/sriov_numvfs
```
If you encounter any error when creating the VF(s), e.g., `tee: '/sys/bus/pci/devices/0000:f7:00.0/sriov_numvfs': No such file or directory`, then try enabling SR-IOV again.
```
# echo 1 | sudo tee /sys/module/vfio_pci/parameters/enable_sriov
```
After you have successfully created the VF, you should see an additional baseband device, in our case, it is `0000:f7:00.1`.
We will use this device with OAI later.
The newly created VF should also be using the same `vfio-pci` driver as the PF, if it is not, you will need to do a `dpdk-devbind.py` to bind it with `vfio-pci`.
```
# sudo dpdk-devbind.py -s
...
Baseband devices using DPDK-compatible driver
=============================================
0000:f7:00.0 'Device 57c0' drv=vfio-pci unused=
0000:f7:00.1 'Device 57c0' drv=vfio-pci unused=
...
```
# Building OAI with ORAN-AAL
OTA deployment is precisely described in the following tutorial:
- [NR_SA_Tutorial_COTS_UE](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/doc/NR_SA_Tutorial_COTS_UE.md)
Instead of section *3.2 Build OAI gNB* from the tutorial, run the following commands:
```bash
# Get openairinterface5g source code
git clone https://gitlab.eurecom.fr/oai/openairinterface5g.git ~/openairinterface5g
cd ~/openairinterface5g
git checkout develop
# Install OAI dependencies
cd ~/openairinterface5g/cmake_targets
./build_oai -I
# Build OAI gNB
cd ~/openairinterface5g
source oaienv
cd cmake_targets
./build_oai -w USRP --ninja --gNB -P --build-lib "ldpc_aal" -C
```
A shared object file `libldpc_aal.so` will be created during the compilation.
This object is conditionally compiled.
The selection of the library to compile is done using `--build-lib ldpc_aal`.
> Note: The required DPDK poll mode driver has to be present on the host machine and required DPDK version has to be installed on the host, prior to building OAI.
# O-RAN AAL DPDK EAL parameters
To configure O-RAN AAL/DPDK BBDEV, you can set the following parameters via the command line of PHY simulators or nr-softmodem:
> Note: the group parameter name has been renamed from `nrLDPC_coding_t2` to
> `nrLDPC_coding_aal` to better reflect that it is a generic AAL accelerator
> card.
- `nrLDPC_coding_aal.dpdk_dev` - **mandatory** parameter, specifies the PCI address of our accelerator. It must follow the format `WWWW:XX:YY.Z`.
- `nrLDPC_coding_aal.dpdk_core_list` - **mandatory** parameter, specifies the CPU cores assigned to DPDK .
Ensure that the CPU cores specified in `nrLDPC_coding_aal.dpdk_core_list` are available and not used by other processes to avoid conflicts.
- `nrLDPC_coding_aal.dpdk_prefix` - optional parameter, DPDK shared data file prefix, by default set to *b6*.
- `nrLDPC_coding_aal.vfio_vf_token` - optional parameter, VFIO token set for the VF, if applicable.
- `nrLDPC_coding_aal.num_harq_codeblock` - optional parameter, size of the HARQ buffer in terms of the number of 32kB blocks, by default set to *512* (maximum for the T2; as for the ACCs, this can be further increased).
- `nrLDPC_coding_aal.is_t2` - optional parameter, set this to 1 when using the Xilinx T2 card.
**Note:** These parameters can also be provided in a configuration file.
Example for the ACC200:
```
nrLDPC_coding_aal : {
dpdk_dev : "0000:f7:00.1";
dpdk_core_list : "14-15";
vfio_vf_token: "00112233-4455-6677-8899-aabbccddeeff";
};
loader : {
ldpc : {
shlibversion : "_aal";
};
};
```
# Running OAI with O-RAN AAL
In general, to offload of the channel coding to the LDPC accelerator, we use use the `--loader.ldpc.shlibversion _aal` option.
Reminder, if you are using the Xilinx T2 card, make sure to set `--nrLDPC_coding_aal.is_t2 1`.
## 5G PHY simulators
### nr_ulsim
Example command:
```bash
cd ~/openairinterface5g
source oaienv
cd cmake_targets/ran_build/build
sudo ./nr_ulsim -n100 -s20 -m20 -r273 -R273 --loader.ldpc.shlibversion _aal --nrLDPC_coding_aal.dpdk_dev 0000:f7:00.1 --nrLDPC_coding_aal.dpdk_core_list 0-1 --nrLDPC_coding_aal.vfio_vf_token 00112233-4455-6677-8899-aabbccddeeff
```
### nr_dlsim
Example command:
```bash
cd ~/openairinterface5g
source oaienv
cd cmake_targets/ran_build/build
sudo ./nr_dlsim -n300 -s30 -R 106 -e 27 --loader.ldpc.shlibversion _aal --nrLDPC_coding_aal.dpdk_dev 0000:f7:00.1 --nrLDPC_coding_aal.dpdk_core_list 0-1 --nrLDPC_coding_aal.vfio_vf_token 00112233-4455-6677-8899-aabbccddeeff
```
## OTA test
### Running OAI gNB with USRP B210/FHI72
Example command:
```bash
cd ~/openairinterface5g
source oaienv
cd cmake_targets/ran_build/build
sudo ./nr-softmodem -O ~/gnb.conf --loader.ldpc.shlibversion _aal --nrLDPC_coding_aal.dpdk_dev 0000:f7:00.1 --nrLDPC_coding_aal.dpdk_core_list 14-15 --nrLDPC_coding_aal.vfio_vf_token 00112233-4455-6677-8899-aabbccddeeff
```
# Known Issue(s)
## BBDEV CPU Usage
When running the E2E setup (this applies to both USRP and FHI72), BBDEV may not be using the list of CPU cores as specified by `nrLDPC_coding_aal.dpdk_core_list` accordingly.
This is an issue under investigation, and subject for future fixes.
In the meantime, we recommend allocating idle, and isolated CPU cores in the configuration for BBDEV.
## Potential Low Throughput
The current implementation has been tested to work in an end-to-end setup and is functional.
However, there are still opportunities for optimization, particularly in LDPC decoding performance, which is an area of ongoing improvement.
As such, downlink/uplink throughput may be suboptimal with the default configurations, but enhancements are actively being explored.
To achieve better E2E performance with the current implementation, we recommend the following adjustments:
1. Increasing the number of LDPC decoding iterations of the L1, e.g., `max_ldpc_iterations` to 200.
2. Increasing the BLER targets of the MAC scheduler.
Example configuration snippet:
```
...
MACRLCs = (
{
num_cc = 1;
tr_s_preference = "local_L1";
tr_n_preference = "local_RRC";
pusch_TargetSNRx10 = 180;
pucch_TargetSNRx10 = 220;
dl_bler_target_upper = .35;
dl_bler_target_lower = .15;
ul_bler_target_upper = .35;
ul_bler_target_lower = .15;
pusch_FailureThres = 1000;
ul_max_mcs = 28;
}
);
L1s = (
{
num_cc = 1;
tr_n_preference = "local_mac";
prach_dtx_threshold = 120;
pucch0_dtx_threshold = 100;
ofdm_offset_divisor = 8; #set this to UINT_MAX for offset 0
max_ldpc_iterations = 200;
}
);
...
```
doc/LDPC_T2_OFFLOAD_SETUP.md
-158
View File
@@ -1,158 +0,0 @@
<table style="border-collapse: collapse; border: none;">
<tr style="border-collapse: collapse; border: none;">
<td style="border-collapse: collapse; border: none;">
<a href="http://www.openairinterface.org/">
<img src="./images/oai_final_logo.png" alt="" border=3 height=50 width=150>
</img>
</a>
</td>
<td style="border-collapse: collapse; border: none; vertical-align: center;">
<b><font size = "5">OAI T1/T2 LDPC offload</font></b>
</td>
</tr>
</table>
**Table of Contents**
[[_TOC_]]
This documentation aims to provide a tutorial for AMD Xilinx T2 Telco card integration into OAI and its usage.
# Requirements
- bitstream image and PMD driver for the T2 card provided by AccelerComm
- DPDK 20.11.9 with patch from Accelercomm: version ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch
- tested on RHEL7.9, RHEL9.2, Ubuntu 22.04
# DPDK setup
## DPDK installation
*Note: Following instructions are valid for ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch version, which is compatible with DPDK 20.11.9. Installation steps, which should be followed for older versions of the patch file (for example ACL_BBDEV_DPDK20.11.3_BL_1006_build_1105_dev_branch_MCT_optimisations_1106_physical_std.patch) are present in older version of this documentation, under the tag 2023.w48.*
```
# Get DPDK source code
git clone https://github.com/DPDK/dpdk-stable.git ~/dpdk-stable
cd ~/dpdk-stable
git checkout v20.11.9
git apply ~/ACL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch
```
Replace `~/ACL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch` by patch file provided by
Accelercomm.
```
cd ~/dpdk-stable
meson setup build
# meson setup --prefix=/opt/dpdk-t2 build for installation with non-default installation prefix
cd build
ninja
sudo ninja install
sudo ldconfig
```
## DPDK configuration
- load required kernel module
```
sudo modprobe vfio-pci
```
- check presence of the card and its PCI addres on the host machine
```
lspci | grep "Xilinx"
```
- bind the card with vfio-pci driver
```
sudo python3 ~/dpdk-stable/usertools/dpdk-devbind.py --bind=vfio-pci 41:00.0
```
Replace PCI address of the card *41:00.0* by address detected by *lspci | grep "Xilinx"* command
- hugepages setup (10 x 1GB hugepages)
```
sudo python3 ~/dpdk-stable/usertools/dpdk-hugepages.py -p 1G --setup 10G
```
*Note: device binding and hugepages setup has to be done after every reboot of
the host machine*
# Modifications in the OAI code
## DPDK lib and PMD path specification
If DPDK library was installed into custom path, you have to point to the right directory with `PKG_CONFIG_PATH`, prior to the OAI build. Sample command to set the DPDK path to `/opt/dpdk-t2/lib64/pkgconfig/`:
```
export PKG_CONFIG_PATH=/opt/dpdk-t2/lib64/pkgconfig/:$PKG_CONFIG_PATH
```
# OAI Build
OTA deployment is precisely described in the following tutorial:
- [NR_SA_Tutorial_COTS_UE](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/doc/NR_SA_Tutorial_COTS_UE.md)
Instead of section *3.2 Build OAI gNB* from the tutorial, run following commands:
```
# Get openairinterface5g source code
git clone https://gitlab.eurecom.fr/oai/openairinterface5g.git ~/openairinterface5g
cd ~/openairinterface5g
git checkout develop
# Install OAI dependencies
cd ~/openairinterface5g/cmake_targets
./build_oai -I
# Build OAI gNB
cd ~/openairinterface5g
source oaienv
cd cmake_targets
./build_oai -w USRP --ninja --gNB -P --build-lib "ldpc_t2" -C
```
Shared object file *libldpc_t2.so* is created during the compilation. This object is conditionally compiled. Selection of the library to compile is done using *--build-lib ldpc_t2*.
*Required poll mode driver has to be present on the host machine and required DPDK version has to be installed on the host, prior to the build of OAI*
# Setup of T2-related DPDK EAL parameters
To configure T2-related DPDK Environment Abstraction Layer (EAL) parameters, you can set the following parameters via the command line of PHY simulators or softmodem:
- `nrLDPC_coding_t2.dpdk_dev` - **mandatory** parameter, specifies PCI address of the T2 card. PCI address of the T2 card can be detected by `lspci | grep "Xilinx"` command.
- `nrLDPC_coding_t2.dpdk_core_list` - **mandatory** parameter, specifies CPU cores assigned to DPDK for T2 processing. Ensure that the CPU cores specified in *nrLDPC_coding_t2.dpdk_core_list* are available and not used by other processes to avoid conflicts.
- `nrLDPC_coding_t2.dpdk_prefix` - DPDK shared data file prefix, by default set to *b6*.
**Note:** These parameters can also be provided in a configuration file:
```
nrLDPC_coding_t2 : {
dpdk_dev : "41:00.0";
dpdk_core_list : "14-15";
};
loader : {
ldpc : {
shlibversion : "_t2";
};
};
```
# 5G PHY simulators
## nr_ulsim test
Offload of the channel decoding to the T2 card is in nr_ulsim specified by *--loader.ldpc.shlibversion _t2* option. Example command for running nr_ulsim with LDPC decoding offload to the T2 card:
```
cd ~/openairinterface5g
source oaienv
cd cmake_targets/ran_build/build
sudo ./nr_ulsim -n100 -s20 -m20 -r273 -R273 --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 01:00.0 --nrLDPC_coding_t2.dpdk_core_list 0-1
```
## nr_dlsim test
Offload of the channel encoding to the AMD Xilinx T2 card is in nr_dlsim specified by *-c* option. Example command for running nr_dlsim with LDPC encoding offload to the T2 card:
```
cd ~/openairinterface5g
source oaienv
cd cmake_targets/ran_build/build
sudo ./nr_dlsim -n300 -s30 -R 106 -e 27 --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 01:00.0 --nrLDPC_coding_t2.dpdk_core_list 0-1
```
# OTA test
Offload of the channel encoding and decoding to the AMD Xilinx T2 card is enabled by *--loader.ldpc.shlibversion _t2* option.
## Run OAI gNB with USRP B210
```
cd ~/openairinterface5g
source oaienv
cd cmake_targets/ran_build/build
sudo ./nr-softmodem -O ../../../targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb.sa.band78.fr1.106PRB.usrpb210.conf --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 01:00.0 --nrLDPC_coding_t2.dpdk_core_list 0-1
```
# Limitations
## AMD Xilinx T2 card
- functionality of the LDPC encoding and decoding offload verified in OTA SISO setup with USRP N310 and Quectel RM500Q, blocking of the card reported for MIMO setup (2 layers)
*Note: AMD Xilinx T1 Telco card is not supported anymore.*
doc/README.md
+1 -1
View File
@@ -64,7 +64,7 @@ There is some general information in the [OpenAirInterface Gitlab Wiki](https://
- [How to run with E2 agent](../openair2/E2AP/README.md)
- [How to run the physical simulators](./physical-simulators.md)
- [How to setup OAI with Nvidia Aerial and Foxconn](./Aerial_FAPI_Split_Tutorial.md)
- [How to setup OAI with AMD T2 Telco card](./LDPC_T2_OFFLOAD_SETUP.md)
- [How to setup OAI with LDPC accelerators (Xilinx T2/Intel ACCs)](./LDPC_OFFLOAD_SETUP.md)
- [How to do a handover](./handover-tutorial.md)
- [How to setup gNB frequency](./gNB_frequency_setup.md)