ldpc_t2: generalize for broader BBDEV support (ACC100/200)
- Generalized LDPC T2 implementation to support other DPDK BBDEV accelerators (e.g., Intel ACC100/200) - Added BBDEV capability checks during initialization - Introduced LLR scaling to improve decoding performance on Intel ACC cards - Implemented self-managed HARQ buffers for ACC200 (no internal HARQ memory) - Library usage documentation update, and added developer documentation Co-authored-by: Romain Beurdouche <romain.beurdouche@eurecom.fr>
This commit is contained in:
Xin Zhe Khooi
committed by
Robert Schmidt
Robert Schmidt
co-authored by
Romain Beurdouche
Romain Beurdouche
parent
b763692792
commit
61ea0a9e81
+244
-63
@@ -7,7 +7,7 @@
|
||||
</a>
|
||||
</td>
|
||||
<td style="border-collapse: collapse; border: none; vertical-align: center;">
|
||||
<b><font size = "5">OAI T1/T2 LDPC offload</font></b>
|
||||
<b><font size = "5">OAI LDPC offload (O-RAN AAL/DPDK BBDEV)</font></b>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
@@ -16,19 +16,57 @@
|
||||
|
||||
[[_TOC_]]
|
||||
|
||||
This documentation aims to provide a tutorial for AMD Xilinx T2 Telco card integration into OAI and its usage.
|
||||
This documentation describes the integration of LDPC coding for lookaside acceleration using O-RAN AAL/DPDK BBDEV in OAI, along with its usage.
|
||||
For details on the implementation, please consult the [developer notes](../openair1/PHY/CODING/nrLDPC_coding/nrLDPC_coding_t2/README.md).
|
||||
|
||||
# Requirements
|
||||
|
||||
- bitstream image and PMD driver for the T2 card provided by AccelerComm
|
||||
- DPDK 20.11.9 with patch from Accelercomm: version ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch
|
||||
- tested on RHEL7.9, RHEL9.2, Ubuntu 22.04
|
||||
In principle, any lookaside LDPC accelerator supporting the O-RAN AAL/DPDK BBDEV should work.
|
||||
However, the current implementation has only been validated for the Xilinx T2, Intel ACC100, and Intel ACC200 (VRB1).
|
||||
Therefore, your mileage may vary when using other BBDEV devices as there may be some hardware-specific changes required -- contributions are welcome!
|
||||
|
||||
# DPDK setup
|
||||
## DPDK Version Requirements
|
||||
|
||||
The following DPDK versions are supported:
|
||||
- For the Xilinx T2 card, DPDK20.11+ is supported.
|
||||
- As for the Intel ACC100/ACC200, only DPDK22.11+ is supported.
|
||||
|
||||
## Tested Devices/ DPDK versions
|
||||
|
||||
### Xilinx T2
|
||||
|
||||
- DPDK20.11.9*.
|
||||
- DPDK22.11.7*.
|
||||
> Note: FPGA bitstream image and the corresponding patch file (e.g., `ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch` for DPDK20.11) from Accelercomm required.
|
||||
|
||||
### Intel ACC100
|
||||
|
||||
- DPDK22.11.7*.
|
||||
- DPDK23.11.3*.
|
||||
- DPDK24.11.2.
|
||||
> Note: [Patch]((https://github.com/DPDK/dpdk/commit/fdde63a1dfc129d0a510a831aa98253b36a2a1cd)) required for pre-DPDK24.11 versions when using the Intel ACC100.
|
||||
|
||||
### Intel ACC200 (also known as VRB1)
|
||||
- DPDK22.11.7.
|
||||
- DPDK23.11.3.
|
||||
- DPDK24.11.2.
|
||||
|
||||
# System Setup
|
||||
## DPDK installation
|
||||
*Note: Following instructions are valid for ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch version, which is compatible with DPDK 20.11.9. Installation steps, which should be followed for older versions of the patch file (for example ACL_BBDEV_DPDK20.11.3_BL_1006_build_1105_dev_branch_MCT_optimisations_1106_physical_std.patch) are present in older version of this documentation, under the tag 2023.w48.*
|
||||
|
||||
```
|
||||
> Important:
|
||||
> - If you are using the Xilinx T2 card, you will need to apply the vendor-supplied patches before compiling DPDK.
|
||||
> - If you are using the Intel ACC100, you will need to [patch](https://github.com/DPDK/dpdk/commit/fdde63a1dfc129d0a510a831aa98253b36a2a1cd) the ACC100's driver if you are using DPDK22.11 or DPDK23.11.
|
||||
|
||||
|
||||
Refer to the guide [here](./ORAN_FHI7.2_Tutorial.md?ref_type=heads#dpdk-data-plane-development-kit) to install, and then validate your DPDK installation.
|
||||
|
||||
<details open>
|
||||
<summary> Notes on DPDK patching/installation for Xilinx T2. </summary>
|
||||
|
||||
*Note: The following instructions apply to `ACCL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch`, compatible with DPDK 20.11.9. For older patches (e.g., `ACL_BBDEV_DPDK20.11.3_BL_1006_build_1105_dev_branch_MCT_optimisations_1106_physical_std.patch`), refer to the T2 documentation in `2023.w48`.*
|
||||
|
||||
```bash
|
||||
# Get DPDK source code
|
||||
git clone https://github.com/DPDK/dpdk-stable.git ~/dpdk-stable
|
||||
cd ~/dpdk-stable
|
||||
@@ -37,50 +75,127 @@ git apply ~/ACL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch
|
||||
```
|
||||
Replace `~/ACL_BBDEV_DPDK20.11.3_ldpc_3.1.918.patch` by patch file provided by
|
||||
Accelercomm.
|
||||
```
|
||||
|
||||
If you would like to install DPDK to a custom directory, here is an example.
|
||||
```bash
|
||||
cd ~/dpdk-stable
|
||||
meson setup build
|
||||
# meson setup --prefix=/opt/dpdk-t2 build for installation with non-default installation prefix
|
||||
# meson setup build
|
||||
meson setup --prefix=/opt/dpdk-t2 build # for installation with non-default installation prefix
|
||||
cd build
|
||||
ninja
|
||||
sudo ninja install
|
||||
sudo ldconfig
|
||||
|
||||
```
|
||||
## DPDK configuration
|
||||
- load required kernel module
|
||||
</details>
|
||||
|
||||
## System configuration
|
||||
|
||||
### Setting up Hugepages
|
||||
|
||||
First, we must setup hugepages on the system.
|
||||
In our setup, we setup 16 of the 1G hugepages.
|
||||
Apart from 1G, 2MB hugepages works too, but make sure to allocate a sufficient number of them.
|
||||
|
||||
```
|
||||
sudo modprobe vfio-pci
|
||||
```
|
||||
- check presence of the card and its PCI addres on the host machine
|
||||
```
|
||||
lspci | grep "Xilinx"
|
||||
```
|
||||
- bind the card with vfio-pci driver
|
||||
```
|
||||
sudo python3 ~/dpdk-stable/usertools/dpdk-devbind.py --bind=vfio-pci 41:00.0
|
||||
```
|
||||
Replace PCI address of the card *41:00.0* by address detected by *lspci | grep "Xilinx"* command
|
||||
- hugepages setup (10 x 1GB hugepages)
|
||||
```
|
||||
sudo python3 ~/dpdk-stable/usertools/dpdk-hugepages.py -p 1G --setup 10G
|
||||
# sudo dpdk-hugepages.py -p 1G --setup 16G
|
||||
```
|
||||
|
||||
*Note: device binding and hugepages setup has to be done after every reboot of
|
||||
the host machine*
|
||||
### Locating the Accelerator
|
||||
|
||||
# Modifications in the OAI code
|
||||
## DPDK lib and PMD path specification
|
||||
If DPDK library was installed into custom path, you have to point to the right directory with `PKG_CONFIG_PATH`, prior to the OAI build. Sample command to set the DPDK path to `/opt/dpdk-t2/lib64/pkgconfig/`:
|
||||
Next, we check whether our system can detect our accelerator using `dpdk-devbind.py`.
|
||||
You should see Baseband devices detected by DPDK, as follows:
|
||||
```
|
||||
export PKG_CONFIG_PATH=/opt/dpdk-t2/lib64/pkgconfig/:$PKG_CONFIG_PATH
|
||||
# sudo dpdk-devbind.py -s
|
||||
...
|
||||
Baseband devices using DPDK-compatible driver
|
||||
=============================================
|
||||
0000:f7:00.0 'Device 57c0' unused=vfio-pci
|
||||
...
|
||||
```
|
||||
|
||||
# OAI Build
|
||||
As you can see here, our Intel ACC200 has the address of `0000:f7:00.0`.
|
||||
Depending on the accelerator you are using, the address may vary.
|
||||
|
||||
### Loading VFIO-PCI and enabling SR-IOV
|
||||
Following, make sure to load the `vfio-pci` kernel modules and ensure that SR-IOV is enabled.
|
||||
|
||||
```
|
||||
# sudo modprobe vfio-pci enable_sriov=1 disable_idle_d3=1
|
||||
```
|
||||
|
||||
### Binding the Accelerator with `vfio-pci`
|
||||
|
||||
Lastly, we bind our accelerator with the `vfio-pci` driver.
|
||||
```
|
||||
# sudo dpdk-devbind.py --bind=vfio-pci 0000:f7:00.0
|
||||
```
|
||||
|
||||
> Note: For the Xilinx T2, we can use this device directly.
|
||||
If you use an Intel vRAN accelerator, read on.
|
||||
|
||||
### Additional Steps for Intel vRAN Accelerators
|
||||
|
||||
> IMPORTANT NOTE:
|
||||
> - Currently, we only support using the Virtual Functions (VFs) of the Intel vRAN accelerators, but not the Physical Function (PF).
|
||||
> - One key advantage of using VFs is that this allows us to share the accelerator with other DU instances on the same machine, which is common in practice.
|
||||
|
||||
If you are using an Intel vRAN accelerator, you will need to use the [pf_bb_config](https://github.com/intel/pf-bb-config) tool to configure the accelerator beforehand.
|
||||
|
||||
#### pf_bb_config
|
||||
For more details, please consult the `pf_bb_config` README.
|
||||
|
||||
```
|
||||
# git clone https://github.com/intel/pf-bb-config
|
||||
# cd ~/pf-bb-config
|
||||
# ./build.sh
|
||||
```
|
||||
This clones and builds the `pf_bb_config` binary.
|
||||
|
||||
Next, we show an example for the Intel ACC200.
|
||||
We use an existing configuration located at `./vrb1/vrb1_config_16vf.cfg`.
|
||||
|
||||
Here, it is necessary to specify a VFIO token (in this case, we use the UUID `00112233-4455-6677-8899-aabbccddeeff`).
|
||||
Note that in practice, a random UUID should be used.
|
||||
```
|
||||
# sudo ./pf_bb_config VRB1 -v 00112233-4455-6677-8899-aabbccddeeff -c vrb1/vrb1_config_16vf.cfg
|
||||
== pf_bb_config Version v25.01-0-g812e032 ==
|
||||
VRB1 PF [0000:f7:00.0] configuration complete!
|
||||
Log file = /var/log/pf_bb_cfg_0000:f7:00.0.log
|
||||
```
|
||||
|
||||
#### Creating VFs
|
||||
|
||||
Finally, we create the VF(s) for our accelerator.
|
||||
In this example, we only create one SR-IOV VF.
|
||||
```
|
||||
# echo 1 | sudo tee /sys/bus/pci/devices/0000:f7:00.0/sriov_numvfs
|
||||
```
|
||||
|
||||
If you encounter any error when creating the VF(s), e.g., `tee: '/sys/bus/pci/devices/0000:f7:00.0/sriov_numvfs': No such file or directory`, then try enabling SR-IOV again.
|
||||
```
|
||||
# echo 1 | sudo tee /sys/module/vfio_pci/parameters/enable_sriov
|
||||
```
|
||||
|
||||
After you have successfully created the VF, you should see an additional baseband device, in our case, it is `0000:f7:00.1`.
|
||||
We will use this device with OAI later.
|
||||
The newly created VF should also be using the same `vfio-pci` driver as the PF, if it is not, you will need to do a `dpdk-devbind.py` to bind it with `vfio-pci`.
|
||||
```
|
||||
# sudo dpdk-devbind.py -s
|
||||
...
|
||||
Baseband devices using DPDK-compatible driver
|
||||
=============================================
|
||||
0000:f7:00.0 'Device 57c0' drv=vfio-pci unused=
|
||||
0000:f7:00.1 'Device 57c0' drv=vfio-pci unused=
|
||||
...
|
||||
```
|
||||
|
||||
# Building OAI with ORAN-AAL
|
||||
OTA deployment is precisely described in the following tutorial:
|
||||
- [NR_SA_Tutorial_COTS_UE](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/doc/NR_SA_Tutorial_COTS_UE.md)
|
||||
Instead of section *3.2 Build OAI gNB* from the tutorial, run following commands:
|
||||
Instead of section *3.2 Build OAI gNB* from the tutorial, run the following commands:
|
||||
|
||||
```
|
||||
```bash
|
||||
# Get openairinterface5g source code
|
||||
git clone https://gitlab.eurecom.fr/oai/openairinterface5g.git ~/openairinterface5g
|
||||
cd ~/openairinterface5g
|
||||
@@ -97,21 +212,35 @@ cd cmake_targets
|
||||
./build_oai -w USRP --ninja --gNB -P --build-lib "ldpc_t2" -C
|
||||
```
|
||||
|
||||
Shared object file *libldpc_t2.so* is created during the compilation. This object is conditionally compiled. Selection of the library to compile is done using *--build-lib ldpc_t2*.
|
||||
A shared object file `libldpc_t2.so` will be created during the compilation.
|
||||
This object is conditionally compiled.
|
||||
The selection of the library to compile is done using `--build-lib ldpc_t2`.
|
||||
|
||||
*Required poll mode driver has to be present on the host machine and required DPDK version has to be installed on the host, prior to the build of OAI*
|
||||
> Note: The required DPDK poll mode driver has to be present on the host machine and required DPDK version has to be installed on the host, prior to building OAI.
|
||||
|
||||
# Setup of T2-related DPDK EAL parameters
|
||||
To configure T2-related DPDK Environment Abstraction Layer (EAL) parameters, you can set the following parameters via the command line of PHY simulators or softmodem:
|
||||
- `nrLDPC_coding_t2.dpdk_dev` - **mandatory** parameter, specifies PCI address of the T2 card. PCI address of the T2 card can be detected by `lspci | grep "Xilinx"` command.
|
||||
- `nrLDPC_coding_t2.dpdk_core_list` - **mandatory** parameter, specifies CPU cores assigned to DPDK for T2 processing. Ensure that the CPU cores specified in *nrLDPC_coding_t2.dpdk_core_list* are available and not used by other processes to avoid conflicts.
|
||||
- `nrLDPC_coding_t2.dpdk_prefix` - DPDK shared data file prefix, by default set to *b6*.
|
||||
# O-RAN AAL DPDK EAL parameters
|
||||
To configure O-RAN AAL/DPDK BBDEV, you can set the following parameters via the command line of PHY simulators or nr-softmodem:
|
||||
|
||||
**Note:** These parameters can also be provided in a configuration file:
|
||||
- `nrLDPC_coding_t2.dpdk_dev` - **mandatory** parameter, specifies the PCI address of our accelerator. It must follow the format `WWWW:XX:YY.Z`.
|
||||
|
||||
- `nrLDPC_coding_t2.dpdk_core_list` - **mandatory** parameter, specifies the CPU cores assigned to DPDK .
|
||||
Ensure that the CPU cores specified in `nrLDPC_coding_t2.dpdk_core_list` are available and not used by other processes to avoid conflicts.
|
||||
|
||||
- `nrLDPC_coding_t2.dpdk_prefix` - optional parameter, DPDK shared data file prefix, by default set to *b6*.
|
||||
|
||||
- `nrLDPC_coding_t2.vfio_vf_token` - optional parameter, VFIO token set for the VF, if applicable.
|
||||
|
||||
- `nrLDPC_coding_t2.num_harq_codeblock` - optional parameter, size of the HARQ buffer in terms of the number of 32kB blocks, by default set to *512* (maximum for the T2; as for the ACCs, this can be further increased).
|
||||
|
||||
- `nrLDPC_coding_t2.eal_init_bbdev` - optional parameter, set this to 1 when using with FHI72 with the Intel ACCs. For the T2, this is not required.
|
||||
|
||||
**Note:** These parameters can also be provided in a configuration file.
|
||||
Example for the ACC200:
|
||||
```
|
||||
nrLDPC_coding_t2 : {
|
||||
dpdk_dev : "41:00.0";
|
||||
dpdk_dev : "0000:f7:00.1";
|
||||
dpdk_core_list : "14-15";
|
||||
vfio_vf_token: "00112233-4455-6677-8899-aabbccddeeff";
|
||||
};
|
||||
|
||||
loader : {
|
||||
@@ -121,38 +250,90 @@ loader : {
|
||||
};
|
||||
```
|
||||
|
||||
# 5G PHY simulators
|
||||
# Running OAI with O-RAN AAL
|
||||
|
||||
## nr_ulsim test
|
||||
Offload of the channel decoding to the T2 card is in nr_ulsim specified by *--loader.ldpc.shlibversion _t2* option. Example command for running nr_ulsim with LDPC decoding offload to the T2 card:
|
||||
```
|
||||
In general, to offload of the channel coding to the LDPC accelerator, we use use the `--loader.ldpc.shlibversion _t2` option.
|
||||
|
||||
## 5G PHY simulators
|
||||
|
||||
### nr_ulsim
|
||||
|
||||
Example command:
|
||||
```bash
|
||||
cd ~/openairinterface5g
|
||||
source oaienv
|
||||
cd cmake_targets/ran_build/build
|
||||
sudo ./nr_ulsim -n100 -s20 -m20 -r273 -R273 --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 01:00.0 --nrLDPC_coding_t2.dpdk_core_list 0-1
|
||||
```
|
||||
## nr_dlsim test
|
||||
Offload of the channel encoding to the AMD Xilinx T2 card is in nr_dlsim specified by *-c* option. Example command for running nr_dlsim with LDPC encoding offload to the T2 card:
|
||||
sudo ./nr_ulsim -n100 -s20 -m20 -r273 -R273 --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 0000:f7:00.1 --nrLDPC_coding_t2.dpdk_core_list 0-1 --nrLDPC_coding_t2.vfio_vf_token 00112233-4455-6677-8899-aabbccddeeff
|
||||
```
|
||||
### nr_dlsim
|
||||
|
||||
Example command:
|
||||
```bash
|
||||
cd ~/openairinterface5g
|
||||
source oaienv
|
||||
cd cmake_targets/ran_build/build
|
||||
sudo ./nr_dlsim -n300 -s30 -R 106 -e 27 --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 01:00.0 --nrLDPC_coding_t2.dpdk_core_list 0-1
|
||||
sudo ./nr_dlsim -n300 -s30 -R 106 -e 27 --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 0000:f7:00.1 --nrLDPC_coding_t2.dpdk_core_list 0-1 --nrLDPC_coding_t2.vfio_vf_token 00112233-4455-6677-8899-aabbccddeeff
|
||||
```
|
||||
|
||||
# OTA test
|
||||
Offload of the channel encoding and decoding to the AMD Xilinx T2 card is enabled by *--loader.ldpc.shlibversion _t2* option.
|
||||
## OTA test
|
||||
|
||||
## Run OAI gNB with USRP B210
|
||||
```
|
||||
### Running OAI gNB with USRP B210/FHI72
|
||||
|
||||
Example command:
|
||||
```bash
|
||||
cd ~/openairinterface5g
|
||||
source oaienv
|
||||
cd cmake_targets/ran_build/build
|
||||
sudo ./nr-softmodem -O ../../../targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb.sa.band78.fr1.106PRB.usrpb210.conf --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 01:00.0 --nrLDPC_coding_t2.dpdk_core_list 0-1
|
||||
sudo ./nr-softmodem -O ~/gnb.conf --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 0000:f7:00.1 --nrLDPC_coding_t2.dpdk_core_list 14-15 --nrLDPC_coding_t2.vfio_vf_token 00112233-4455-6677-8899-aabbccddeeff --nrLDPC_coding_t2.eal_init_bbdev 1
|
||||
```
|
||||
|
||||
# Limitations
|
||||
## AMD Xilinx T2 card
|
||||
- functionality of the LDPC encoding and decoding offload verified in OTA SISO setup with USRP N310 and Quectel RM500Q, blocking of the card reported for MIMO setup (2 layers)
|
||||
# Known Issue(s)
|
||||
|
||||
*Note: AMD Xilinx T1 Telco card is not supported anymore.*
|
||||
## BBDEV CPU Usage
|
||||
|
||||
When running the E2E setup (this applies to both USRP and FHI72), BBDEV may not be using the list of CPU cores as specified by `nrLDPC_coding_t2.dpdk_core_list` accordingly.
|
||||
This is an issue under investigation, and subject for future fixes.
|
||||
In the meantime, we recommend allocating idle, and isolated CPU cores in the configuration for BBDEV.
|
||||
|
||||
## Potential Low Throughput
|
||||
|
||||
The current implementation has been tested to work in an end-to-end setup and is functional.
|
||||
However, there are still opportunities for optimization, particularly in LDPC decoding performance, which is an area of ongoing improvement.
|
||||
As such, downlink/uplink throughput may be suboptimal with the default configurations, but enhancements are actively being explored.
|
||||
|
||||
To achieve better E2E performance with the current implementation, we recommend the following adjustments:
|
||||
1. Increasing the number of LDPC decoding iterations of the L1, e.g., `max_ldpc_iterations` to 200.
|
||||
2. Increasing the BLER targets of the MAC scheduler.
|
||||
|
||||
Example configuration snippet:
|
||||
```
|
||||
...
|
||||
MACRLCs = (
|
||||
{
|
||||
num_cc = 1;
|
||||
tr_s_preference = "local_L1";
|
||||
tr_n_preference = "local_RRC";
|
||||
pusch_TargetSNRx10 = 180;
|
||||
pucch_TargetSNRx10 = 220;
|
||||
dl_bler_target_upper = .35;
|
||||
dl_bler_target_lower = .15;
|
||||
ul_bler_target_upper = .35;
|
||||
ul_bler_target_lower = .15;
|
||||
pusch_FailureThres = 1000;
|
||||
ul_max_mcs = 28;
|
||||
|
||||
}
|
||||
);
|
||||
|
||||
L1s = (
|
||||
{
|
||||
num_cc = 1;
|
||||
tr_n_preference = "local_mac";
|
||||
prach_dtx_threshold = 120;
|
||||
pucch0_dtx_threshold = 100;
|
||||
ofdm_offset_divisor = 8; #set this to UINT_MAX for offset 0
|
||||
max_ldpc_iterations = 200;
|
||||
}
|
||||
);
|
||||
...
|
||||
```
|
||||
|
||||
@@ -39,7 +39,7 @@ Building C object CMakeFiles/ldpc_t2.dir/openair1/PHY/CODING/nrLDPC_coding/nrLDP
|
||||
Linking C shared module libldpc_t2.so
|
||||
```
|
||||
|
||||
At runtime, to successfully use the T2 board, you need to install vendor specific drivers and tools.\
|
||||
At runtime, to successfully use LDPC accelerators (e.g., Xilinx T2/Intel ACCs), you will need to install the corresponding drivers and tools.
|
||||
Please refer to the dedicated documentation at [LDPC_T2_OFFLOAD_SETUP.md](file://../../../../doc/LDPC_T2_OFFLOAD_SETUP.md).
|
||||
|
||||
`./nr-softmodem -O libconfig:gnb.band78.sa.fr1.106PRB.usrpb210.conf:dbgl5 --rfsim --rfsimulator.serveraddr server --log_config.gtpu_log_level info --loader.ldpc.shlibversion _t2 --nrLDPC_coding_t2.dpdk_dev 01:00.0 --nrLDPC_coding_t2.dpdk_core_list 0-1`
|
||||
|
||||
@@ -1,25 +1,35 @@
|
||||
##########################################################
|
||||
|
||||
# LDPC offload library - AMD T2 Accelerator Card
|
||||
# LDPC offload library - DPDK BBDEV
|
||||
##########################################################
|
||||
|
||||
add_boolean_option(ENABLE_LDPC_T2 OFF "Build support for LDPC Offload to T2 library" OFF)
|
||||
|
||||
if (ENABLE_LDPC_T2)
|
||||
|
||||
pkg_check_modules(LIBDPDK_T2 REQUIRED libdpdk>=20.11.9)
|
||||
|
||||
if(LIBDPDK_T2_VERSION VERSION_LESS "21.11")
|
||||
add_compile_definitions(DPDK_VER_PRE_21_11)
|
||||
endif()
|
||||
|
||||
find_library(PMD_T2 NAMES rte_baseband_accl_ldpc HINTS ${LIBDPDK_T2_LIBRARY_DIRS})
|
||||
if (NOT PMD_T2)
|
||||
message(FATAL_ERROR "could not find poll-mode driver for AccelerComm T2 LDPC Offload (rte_baseband_accl_ldpc.so)")
|
||||
if (PMD_T2)
|
||||
add_compile_definitions(LDPC_T2)
|
||||
message(STATUS "found poll-mode driver for AccelerComm T2 LDPC Offload (rte_baseband_accl_ldpc.so)")
|
||||
else()
|
||||
if (LIBDPDK_T2_VERSION VERSION_LESS "22.11")
|
||||
message(FATAL_ERROR "for non-T2 BBDEV PMDs, we support the Intel ACC100 and ACC200 (VRB1), using DPDK 22.11 or later.")
|
||||
endif()
|
||||
endif()
|
||||
|
||||
message(STATUS "T2 build: use ${PMD_T2}")
|
||||
add_library(ldpc_t2 MODULE nrLDPC_coding_t2.c)
|
||||
|
||||
set_target_properties(ldpc_t2 PROPERTIES COMPILE_FLAGS "-DALLOW_EXPERIMENTAL_API")
|
||||
set_target_properties(ldpc_t2 PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})
|
||||
target_link_libraries(ldpc_t2 PRIVATE ldpc_gen_HEADERS ${LIBDPDK_T2_LDFLAGS} ${PMD_T2})
|
||||
target_include_directories(ldpc_t2 PRIVATE ${LIBDPDK_T2_INCLUDE_DIRS})
|
||||
target_link_libraries(ldpc_t2 PRIVATE ldpc_gen_HEADERS ${LIBDPDK_T2_LDFLAGS})
|
||||
if (PMD_T2)
|
||||
target_link_libraries(ldpc_t2 PRIVATE ${PMD_T2})
|
||||
endif()
|
||||
|
||||
add_dependencies(nr-softmodem ldpc_t2)
|
||||
add_dependencies(nr-uesoftmodem ldpc_t2)
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
This document highlights some of the key aspects of the implementation for O-RAN AAL/DPDK BBDEV; for user documentation, please consult the [setup guide](../../../../../doc/LDPC_T2_OFFLOAD_SETUP.md) instead.
|
||||
|
||||
## Overview
|
||||
|
||||
The current implementation supports the following BBDEV devices and respective DPDK versions:
|
||||
- Xilinx T2 on DPDK20.11+
|
||||
- Intel ACC100* on DPDK22.11+
|
||||
- Intel ACC200 (VRB1) on DPDK22.11+
|
||||
|
||||
> Important notes:
|
||||
> - For the Intel ACC100, for DPDK22.11 and DPDK23.11, users are required to [patch](https://github.com/DPDK/dpdk/commit/fdde63a1dfc129d0a510a831aa98253b36a2a1cd) the ACC100's driver.
|
||||
> - If you intend to use this implementation in an FHI7.2 setup, only the F-release and beyond is supported.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
Below, we highlight some of the important details in this implementation.
|
||||
|
||||
### HARQ
|
||||
|
||||
To maintain the HARQ buffers in the LDPC decoding process, the BBDEV either uses its internal memory (not visible to DPDK), or using external memory (i.e., DDR) managed by OAI.
|
||||
For the Xilinx T2 and Intel ACC100, they use the internal memory while the Intel ACC200 uses the DDR memory.
|
||||
|
||||
If internal memory is supported by the device, we will set `RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_IN_ENABLE` and `RTE_BBDEV_LDPC_INTERNAL_HARQ_MEMORY_OUT_ENABLE`.
|
||||
|
||||
The following two fields are important for HARQ: `harq_combined_input` and `harq_combined_output`.
|
||||
|
||||
For the first transmission, there is no HARQ input.
|
||||
Thus, the flag `RTE_BBDEV_LDPC_HQ_COMBINE_IN_ENABLE` is not set.
|
||||
However, there will be a HARQ output, so we set the flag `RTE_BBDEV_LDPC_HQ_COMBINE_OUT_ENABLE`.
|
||||
We also set the corresponding fields for `harq_combined_output`.
|
||||
|
||||
For the second transmission (or subsequent retransmissions), we need to provide a HARQ input at the same time, there will also be a HARQ output.
|
||||
Therefore, both fields, `harq_combined_input` and `harq_combined_output` must be set accordingly.
|
||||
|
||||
As HARQ requires the output of the previous transmission/ HARQ round as its input, we need to maintain a separate variable to hold the previous round's information.
|
||||
To do so, we define `harq_buffers` (which is a pointer to an array of size `num_harq_codeblock` allocated in the heap later) in the `active_device` struct (see the following) in our implementation.
|
||||
```C
|
||||
struct active_device {
|
||||
...
|
||||
uint32_t num_harq_codeblock;
|
||||
/* Persistent data structure to keep track of HARQ-related information */
|
||||
// Note: This is used to store/keep track of the combined output information across iterations
|
||||
struct rte_bbdev_op_data *harq_buffers;
|
||||
...
|
||||
} active_dev;
|
||||
```
|
||||
|
||||
|
||||
In the following, we elaborate on how the HARQ-related fields are set.
|
||||
|
||||
#### Internal Memory
|
||||
|
||||
When using the internal memory, we need to specify the *offset* of the hardware's memory where we want to store the HARQ combined output, and read from it accordingly later.
|
||||
|
||||
In the current implementation, we use a fixed offset of 32K, i.e., this means that each block is 32K.
|
||||
|
||||
An index is derived from a unique HARQ PID assigned by OAI, modulo `active_dev.num_harq_codeblock` (which is configurable by the user through `nrLDPC_coding_t2.num_harq_codeblock`).
|
||||
Using that index, we then derived the corresponding offset in the memory (i.e., `harq_combined_offset`).
|
||||
|
||||
```C
|
||||
...
|
||||
|
||||
// Calculate offset in the HARQ combined buffers
|
||||
// Unique segment offset
|
||||
uint32_t segment_offset = (nrLDPC_slot_decoding_parameters->TBs[h].harq_unique_pid * NR_LDPC_MAX_NUM_CB) + i;
|
||||
// Prune to avoid shooting above maximum id
|
||||
uint32_t pruned_segment_offset = segment_offset % active_dev.num_harq_codeblock;
|
||||
// Segment offset to byte offset
|
||||
uint32_t harq_combined_offset = pruned_segment_offset * LDPC_MAX_CB_SIZE;
|
||||
|
||||
...
|
||||
```
|
||||
|
||||
This `harq_combined_offset` is then set for `harq_combined_input.offset` and `harq_combined_output.offset` to tell BBDEV where to read/write the HARQ buffers.
|
||||
|
||||
Especially for the Intel ACCs, when providing the HARQ input, the `harq_combined_input.length` must be provided.
|
||||
To do so, we maintain the length of previous round's output using `harq_buffers` by copying it in `retrieve_ldpc_dec_op`.
|
||||
|
||||
#### External Memory
|
||||
|
||||
Different from the Xilinx T2 and Intel ACC100, the Intel ACC200 uses DDR memory to maintain the HARQ buffers which is managed by OAI.
|
||||
|
||||
In short, we use `harq_buffers` from the `active_device` struct, which have been initialzed with corresponding DPDK mempools for BBDEV to read/write the HARQ input/outputs.
|
||||
|
||||
For indexing, we use the `pruned_segment_offset`, which is derived from the unique HARQ PID assigned by OAI, modulo `active_dev.num_harq_codeblock` (which is configurable by the user through `nrLDPC_coding_t2.num_harq_codeblock`).
|
||||
|
||||
`harq_combined_input.offset` and `harq_combined_output.offset` is always set to 0 in our implementation.
|
||||
Instead, we provide BBDEV the pointers to our allocated mempool regions through `harq_combined_input.data` and `harq_combined_output.data`.
|
||||
|
||||
In `retrieve_ldpc_dec_op`, we perform a `rte_memcpy` to copy the HARQ combined outputs to `harq_buffers`, along with other necessary metadata information to prepare for subsequent HARQ rounds.
|
||||
Similar to the Intel ACC100, when providing the HARQ input, the `harq_combined_input.length` must be provided.
|
||||
|
||||
### LLR Scaling
|
||||
|
||||
The LLR output from OAI's demodulation block is in 16 bits.
|
||||
However, accelerators typically assume that the LLR inputs are scaled down using fixed-point arithmetic, e.g., 8 bits.
|
||||
|
||||
For the Xilinx T2, saturation is performed.
|
||||
However, our tests have shown that this implementation did not work well for the Intel ACCs.
|
||||
|
||||
To that end, we perform LLR scaling differently for the Intel ACCs.
|
||||
Through the function `llr_scaling`, we look at the min and max LLR values, perform scaling and subsequently saturation accordingly.
|
||||
|
||||
The fixed-point representations supported by the hardware is defined by the `llr_size` and `llr_decimal`.
|
||||
For example, if `llr_size=8`, and `llr_decimal=1`, it means that we use a S8.1 representation (1 fractional bit), the min and max values are -64 to 63.5.
|
||||
|
||||
> *Limitation:*
|
||||
> We observe that in certain configurations, such as when using 2 MIMO layers, the LLR (Log-Likelihood Ratio) outputs exhibit a much wider dynamic range (e.g., from -10,000 to 10,000) compared to the typical sub-1,000 range seen in most other cases.
|
||||
> This suggests potential inconsistencies in the LLR values generated by the demodulator.
|
||||
> We believe that the decoding performance, when compared to the CPU implementation under the same number of LDPC decoding iterations, could be improved by refining the LLR scaling approach or revisiting the LLR generation process during demodulation.
|
||||
> This remains an area of active investigation.
|
||||
>
|
||||
> *Workaround:*
|
||||
> As a temporary measure, increasing the number of LDPC decoding iterations—e.g., to 150—has shown to yield a decoding success rate comparable to that of the CPU implementation.
|
||||
> This comes at a negligible increase in decoding time based on our current evaluation.
|
||||
|
||||
### Special Case(s)
|
||||
|
||||
#### TB mode
|
||||
|
||||
For the T2, only CB mode is supported.
|
||||
|
||||
For the Intel ACCs, while we also use CB mode mostly, there is a special case that must be handled properly as specified by the BBDEV documentations:
|
||||
```
|
||||
... The case when one CB belongs to TB and is being enqueued individually to BBDEV, this case is considered as a special case of partial TB where its number of CBs is 1. Therefore, it requires to get processed in TB-mode. ...
|
||||
```
|
||||
|
||||
In such a case, we perform a check on whether there is only one TB, AND if the TB has only one CB.
|
||||
If so, this has to be handled separately by using TB mode.
|
||||
|
||||
### Future Work(s)
|
||||
|
||||
#### TB mode
|
||||
|
||||
The current channel coding interface assumes LDPC encoding and decoding are performed on a per-code block basis.
|
||||
However, to fully leverage Transport Block (TB) mode as supported in BBDEV, which can improve efficiency, the L1 implementation surrounding channel coding needs to be refactored.
|
||||
This remains an area for future investigation.
|
||||
|
||||
#### CRC24A/CRC16
|
||||
|
||||
CRC24A/CRC16 is supported by the BBDEV device (i.e., Intel ACC200).
|
||||
However, we do not make use of them and rely on OAI's existing CRC functions.
|
||||
To use of BBDEV's CRC24A/CRC16 capability will be looked into in future MRs.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -2,8 +2,8 @@
|
||||
* Copyright(c) 2017 Intel Corporation
|
||||
*/
|
||||
|
||||
#ifndef _MAIN_H_
|
||||
#define _MAIN_H_
|
||||
#ifndef __NRLDPC_CODING_T2_H__
|
||||
#define __NRLDPC_CODING_T2_H__
|
||||
|
||||
#include <stddef.h>
|
||||
#include <sys/queue.h>
|
||||
@@ -17,6 +17,7 @@
|
||||
enum op_data_type {
|
||||
DATA_INPUT = 0,
|
||||
DATA_HARD_OUTPUT,
|
||||
DATA_HARQ_OUTPUT,
|
||||
DATA_NUM_TYPES,
|
||||
};
|
||||
|
||||
|
||||
Reference in New Issue
Block a user