From 283063f3da80bee0d419ae7b5746349f2b4a42dd Mon Sep 17 00:00:00 2001 From: Reem Bahsoun Date: Tue, 27 Jan 2026 14:13:26 +0100 Subject: [PATCH] Update tutorial to align with the new Aerial release (25-3) --- doc/Aerial_FAPI_Split_Tutorial.md | 294 +++++++++++++++++++----------- 1 file changed, 190 insertions(+), 104 deletions(-) diff --git a/doc/Aerial_FAPI_Split_Tutorial.md b/doc/Aerial_FAPI_Split_Tutorial.md index 58380ba611..24a085c8e4 100644 --- a/doc/Aerial_FAPI_Split_Tutorial.md +++ b/doc/Aerial_FAPI_Split_Tutorial.md @@ -1,4 +1,4 @@ -# OAI 7.2 Fronthaul Interface 5G SA Tutorial +# OAI - Aerial FAPI Split Tutorial **Table of Contents** @@ -14,9 +14,12 @@ The hardware on which we have tried this tutorial: | Dell PowerEdge R750 (Dual Intel Xeon Gold 6336Y CPU @ 2.4G, 24C/48T (185W), 512GB RDIMM, 3200MT/s) |Ubuntu 22.04.3 LTS (5.15.0-72-lowlatency)| NVIDIA Converged Accelerator A100X (24.39.2048) | | Supermicro Grace Hopper MGX ARS-111GL-NHR (Neoverse-V2, 3.4GHz, 72C/72T, 576GB LPDDR5) | Ubuntu 22.04.5 LTS (6.5.0-1019-nvidia-64k) |NVIDIA BlueField3 (32.41.1000)| -**NOTE**: -- These are not minimum hardware requirements. This is the configuration of our servers. The NIC card should support hardware PTP time stamping. -- Starting from tag [2025.w13](https://gitlab.eurecom.fr/oai/openairinterface5g/-/tree/2025.w13?ref_type=tags) of OAI, we are only testing with the Grace Hopper server. +**Note**: +- These are not minimum hardware requirements. This is the configuration of our + servers. The NIC card should support hardware PTP time stamping. +- Starting from tag + [2025.w13](https://gitlab.eurecom.fr/oai/openairinterface5g/-/tree/2025.w13?ref_type=tags) + of OAI, we are only testing with the Grace Hopper server. PTP enabled switches and grandmaster clock we have tested with: @@ -54,10 +57,10 @@ The UEs that have been tested and confirmed working with Aerial are the followin To set up the L1 and install the components manually refer to this [instructions page](https://docs.nvidia.com/aerial/cuda-accelerated-ran/index.html). **Note**: -- As of wk36, the L1 must be compiled with the following CMake flag: `-DSCF_FAPI_10_04_SRS=ON` , this is due to the usage -of the FAPI 10.04 version of the SRS PDU, and RX_Beamforming PDU. -- To configure the Gigabyte server please refer to these [instructions](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/2025.w13/doc/Aerial_FAPI_Split_Tutorial.md) -- The last release to support the Gigabyte server is **Aerial CUDA-Accelerated RAN 24-1**. +- To configure the Gigabyte server please refer to these + [instructions](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/2025.w13/doc/Aerial_FAPI_Split_Tutorial.md) +- The last release to support the Gigabyte server is **Aerial CUDA-Accelerated + RAN 24-1**. #### CPU allocation @@ -67,21 +70,43 @@ of the FAPI 10.04 version of the SRS PDU, and RX_Beamforming PDU. **Grace Hopper MGX ARS-111GL-NHR** -| Applicative Threads | Allocated CPUs | -|------------------------|----------------| -| PTP & PHC2SYS Services | 41 | -| OAI `nr-softmodem` | 13-16 | +| Applicative Threads | Allocated CPUs | +|--------------------------------------|--------------------------------| +| `workers_ul` | 4, 5 | +| `workers_dl` | 6, 7, 8 | +| `timer_thread_config` | 9 | +| `message_thread_config` | 9 | +| `pcap_shm_caching_cpu_core` | 10 | +| `pcap_file_saving_cpu_core` | 10 | +| `dpdk_thread` | 10 | +| `ul_pcap_capture_thread_cpu_affinity`| 10 | +| `pcap_logger_thread_cpu_affinity` | 10 | +| `h2d_copy_thread_cpu_affinity` | 11 | +| `fh_stats_dump_cpu_core` | -1 | +| `pdump_client_thread` | -1 | +| `debug_worker` | -1 | +| `prometheus_thread` | -1 | +| | | +| OAI `nr-softmodem` | 13,14,15,16 | -### PTP configuration +**Note**: +- `-1` indicates that no explicit CPU pinning is applied and the thread +is scheduled by the Linux kernel. +- core 10 is a `low_priority_core` and it is shared by all low-priority threads. -1. You need to install the `linuxptp` debian package. It will install both ptp4l and phc2sys. +#### PTP configuration + +1. Install the `linuxptp` debian package. It will install both + ptp4l and phc2sys. ```bash #Ubuntu sudo apt install linuxptp -y ``` -Once installed you can use this configuration file for ptp4l (`/etc/ptp4l.conf`). Here the clock domain is 24 so you can adjust it according to your PTP GM clock domain +Once installed you can use this configuration file for ptp4l +(`/etc/ptp4l.conf`). Here the clock domain is 24 so you can adjust it according +to your PTP GM clock domain ``` [global] @@ -94,25 +119,27 @@ tx_timestamp_timeout 30 network_transport L2 ``` +The service of ptp4l (`/lib/systemd/system/ptp4l.service`) should be configured +as below: -The service of ptp4l (`/lib/systemd/system/ptp4l.service`) should be configured as below: ``` -[Unit] -Description=Precision Time Protocol (PTP) service +[Unit] Description=Precision Time Protocol (PTP) service Documentation=man:ptp4l [Service] Restart=always RestartSec=5s Type=simple -ExecStart=taskset -c 41 /usr/sbin/ptp4l -f /etc/ptp.conf +ExecStart=/usr/sbin/ptp4l -f /etc/ptp.conf [Install] WantedBy=multi-user.target ``` -and service of phc2sys (`/lib/systemd/system/phc2sys.service`) should be configured as below: +and service of phc2sys (`/lib/systemd/system/phc2sys.service`) should be +configured as below: + ``` [Unit] Description=Synchronize system clock or PTP hardware clock (PHC) @@ -131,10 +158,25 @@ ExecStart=/usr/sbin/phc2sys -a -r -n 24 WantedBy=multi-user.target ``` +**Note**: As of release 25-3 there's no need to allocate a core for PTP. + +## Prepare the L1 + +- Follow these + [instructions](https://github.com/NVIDIA/aerial-cuda-accelerated-ran) to + clone the `aerial-cuda-accelerated-ran` repository, pull and build the cuBB + image. +- The CMake flags we use to compile the L1 are: + - `-DSCF_FAPI_10_04_SRS=ON` this flag must be used as of OAI tag `2025.w36` + and this is due to the usage of the FAPI 10.04 version of the SRS PDU, and + RX_Beamforming PDU. + - `-DENABLE_CONFORMANCE_TM_PDSCH_PDCCH=OFF` this option should only be + enabled when doing conformance testing with testmac. ## Build OAI gNB If it's not already cloned, the first step is to clone OAI repository + ```bash git clone https://gitlab.eurecom.fr/oai/openairinterface5g.git ~/openairinterface5g cd ~/openairinterface5g/ @@ -142,65 +184,42 @@ cd ~/openairinterface5g/ ### Get nvIPC sources from the L1 container -The library used for communication between L1 and L2 components is called nvIPC, and is developed by NVIDIA. It is not open-source and can't be freely distributed. -In order to achieve this communication, we need to obtain the nvIPC source files from the L1 container (cuBB) and place it in the gNB project directory `~/openairinterface5g`. -This allows us to build and install this library when building the L2 docker container. - -Check whether your L1 container is running: -```bash -~$ docker container ls -a -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -a9681a0c4a10 14dca2002237 "/opt/nvidia/nvidia_…" 3 months ago Exited (137) 10 days ago cuBB - -``` -If it is not running, you may start it and logging into the container by running the following: -```bash -~$ docker start cuBB -cuBB -~$ docker exec -it cuBB bash -aerial@c_aerial_aerial:/opt/nvidia/cuBB# -``` - -After logging into the container, you need to pack the nvIPC sources and copy them to the host ( the command creates a `tar.gz` file with the following name format: `nvipc_src.YYYY.MM.DD.tar.gz`) -```bash -~$ docker exec -it cuBB bash -aerial@c_aerial_aerial:/opt/nvidia/cuBB# cd cuPHY-CP/gt_common_libs -aerial@c_aerial_aerial:/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs#./pack_nvipc.sh -nvipc_src.YYYY.MM.DD/ -... ---------------------------------------------- -Pack nvipc source code finished: -/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs/nvipc_src.YYYY.MM.DD.tar.gz -aerial@c_aerial_aerial:/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs# sudo cp nvipc_src.yyyy.mm.dd.tar.gz /opt/cuBB/share/ -aerial@c_aerial_aerial:/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs# exit -``` - -The file should now be present in the `~/openairinterface5g/ci-scripts/yaml_files/sa_gnb_aerial/` directory, from where it is moved into `~/openairinterface5g` -```bash -~$ mv ~/openairinterface5g/ci-scripts/yaml_files/sa_gnb_aerial/nvipc_src.YYYY.MM.DD.tar.gz ~/openairinterface5g/ -``` - -Alternatively, after running `./pack_nvipc.sh` , you can exit the container and copy `nvipc_src.YYYY.MM.DD.tar.gz` using `docker cp` +The library used for communication between L1 and L2 components is called +nvIPC, and is developed by NVIDIA. In order to achieve this communication, we +need to obtain the nvIPC source files from the L1 container (cuBB) and place it +in the gNB project directory `~/openairinterface5g`. This allows us to build +and install this library when building the L2 docker image. ```bash -~$ docker exec -it cuBB bash -aerial@c_aerial_aerial:/opt/nvidia/cuBB# cd cuPHY-CP/gt_common_libs -aerial@c_aerial_aerial:/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs#./pack_nvipc.sh -nvipc_src.YYYY.MM.DD/ -... ---------------------------------------------- -Pack nvipc source code finished: -/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs/nvipc_src.YYYY.MM.DD.tar.gz -aerial@c_aerial_aerial:/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs# exit -~$ docker cp nv-cubb:/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs/nvipc_src.YYYY.MM.DD.tar.gz ~/openairinterface5g +# Start interactive development container +./cuPHY-CP/container/run_aerial.sh + +# Pack the nvIPC sources and copy them to the host ( the command creates a `tar.gz` file with the following name format: `nvipc_src.YYYY.MM.DD.tar.gz`) +aerial@c_aerial_oaicicd:/opt/nvidia/cuBB# cd cuPHY-CP/gt_common_libs +aerial@c_aerial_oaicicd:/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs#./pack_nvipc.sh +nvipc_src.YYYY.MM.DD/ ... --------------------------------------------- Pack +nvipc source code finished:/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs/nvipc_src.YYYY.MM.DD.tar.gz ``` -*Important note:* For using docker cp, make sure to copy the entire name of the created nvipc_src tar.gz file. + +Keep the interactive development container open and in another terminal use +`docker cp` to copy the library + +```bash +docker cp c_aerial_oaicicd:/opt/nvidia/cuBB/cuPHY-CP/gt_common_libs/nvipc_src.YYYY.MM.DD.tar.gz ~/openairinterface/ +``` +*Important note:* For using docker cp, make sure to +copy the entire name of the created nvipc_src tar.gz file. + +You can now exit the interactive development container. + +With the nvIPC sources in the project directory, the L2 docker image can be built. -With the nvIPC sources in the project directory, the docker image can be built. ### Building OAI gNB docker image -In order to build the target image (`oai-gnb-aerial`), first you should build a common shared image (`ran-base`) +In order to build the target image (`oai-gnb-aerial`), first you should build a +common shared image (`ran-base`) + ```bash ~$ cd ~/openairinterface5g/ ~/openairinterface5g$ docker build . -f docker/Dockerfile.base.ubuntu --tag ran-base:latest @@ -210,52 +229,113 @@ In order to build the target image (`oai-gnb-aerial`), first you should build a ## Running the setup -In order to use Docker Compose to automatically start and stop the setup, we first need to create a pre-built image of L1 after compiling the L1 software and making the necessary adjustments to its configuration files. -The process of preparing L1 is covered in NVIDIA's documentation and falls outside the scope of this document. - -### Prepare the L1 image -After preparing the L1 software, the container needs to be committed to create an image where L1 is ready for execution, which can later be referenced by a `docker-compose.yaml` file. - -*Note:* The default L1 configuration file is `cuphycontroller_P5G_FXN_GH.yaml`, located in `/opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/`. -In this file the RU MAC address needs to be specified before commiting the image. - -```bash -~$ docker commit nv-cubb cubb-build:25-2.1 -~$ docker image ls -.. -cubb-build 25-2.1 824156e0334c 2 weeks ago 23.9GB --.. -``` - ### Adapt the OAI-gNB configuration file to your system/workspace -Edit the sample OAI gNB configuration file and check following parameters: +Edit the [OAI gNB configuration file](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/ci-scripts/conf_files/gnb-vnf.sa.band78.273prb.aerial.conf?ref_type=heads) +and check the following parameters: * `gNBs` section * The PLMN section shall match the one defined in the AMF + * To calculate the `absoluteFrequencySSB` and `dl_absoluteFrequencyPointA`, + please follow these + [instructions](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/doc/gNB_frequency_setup.md) * `amf_ip_address` shall be the correct AMF IP address in your system * `GNB_IPV4_ADDRESS_FOR_NG_AMF` shall match your DU N2 interface IP address * `GNB_IPV4_ADDRESS_FOR_NGU` shall match your DU N3 interface IP address -The default amf_ip_address:ipv4 value is 192.168.70.132, when installing the CN5G following [this tutorial](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/doc/NR_SA_Tutorial_OAI_CN5G.md) -Both 'GNB_IPV4_ADDRESS_FOR_NG_AMF' and 'GNB_IPV4_ADDRESS_FOR_NGU' need to be set to the IP address of the NIC referenced previously. +The default amf_ip_address:ipv4 value is 192.168.70.132, when installing the +CN5G following [this +tutorial](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/doc/NR_SA_Tutorial_OAI_CN5G.md) +Both `GNB_IPV4_ADDRESS_FOR_NG_AMF` and `GNB_IPV4_ADDRESS_FOR_NGU` need to be +set to the IP address of the NIC referenced previously. +**Note**: If the Core Network is running on the same server, 3 cores should be +allocated to it. 2 for the UPF and 1 for the all the remaining services as shown +below. -### Running docker compose -#### Aerial L1 entrypoint script -The `aerial_l1_entrypoint` script is used by the L1 container to start the L1 software and is called in the Docker Compose file. -It begins by setting up environment variables, restarting NVIDIA MPS, and finally running `cuphycontroller_scf`. +```patch +diff --git a/docker-compose/docker-compose-basic-nrf.yaml b/docker-compose/docker-compose-basic-nrf.yaml +index 7fbefc9..d5ef964 100644 +--- a/docker-compose/docker-compose-basic-nrf.yaml ++++ b/docker-compose/docker-compose-basic-nrf.yaml +@@ -28,6 +28,7 @@ services: + - 8080/tcp + volumes: + - ./conf/basic_nrf_config.yaml:/openair-udr/etc/config.yaml ++ cpuset: "15" + environment: + - TZ=Europe/Paris + depends_on: +``` -The L1 software is executed with an argument that specifies which configuration file to use. If not modified, the default argument is set to `P5G_FXN_GH`. +### Aerial L1 entrypoint script -After building the gNB image, and preparing the configuration file, the setup can be run with the following command: +The `aerial_l1_entrypoint` script is used by the L1 container to start the L1 +software and is mounted in the Docker Compose file. It begins by setting up +environment variables, restarting NVIDIA MPS, and finally running +`cuphycontroller_scf`. + +The L1 software is executed with an argument that specifies which configuration +file to use. If not modified, the default argument is set to `P5G_WNC_GH`. + +### Aerial L1 configuration + +```bash +cd ~/aerial-cuda-accelerated-ran/cuPHY-CP/cuphycontroller/config/ +``` + +Choose the configuration file that matches your setup and make the necessary +changes. We currently use `cuphycontroller_P5G_WNC_GH.yaml`. + +Make sure that the following parameters are modified: + +``` +dst_mac_addr: +nic: +vlan: +``` + +### Docker compose + +We recommend the user to apply the patch below for `docker-compose.yaml`. +```patch +diff --git a/ci-scripts/yaml_files/sa_gnb_aerial/docker-compose.yaml b/ci-scripts/yaml_files/sa_gnb_aerial/docker-compose.yaml +index 985fe9a6a3..0774d826ac 100644 +--- a/ci-scripts/yaml_files/sa_gnb_aerial/docker-compose.yaml ++++ b/ci-scripts/yaml_files/sa_gnb_aerial/docker-compose.yaml +@@ -15,10 +15,10 @@ services: + stdin_open: true + tty: true + volumes: ++ - ~/aerial-cuda-accelerated-ran:/opt/nvidia/cuBB/ + - /lib/modules:/lib/modules + - /dev/hugepages:/dev/hugepages + - /usr/src:/usr/src +- - ./cuphycontroller_P5G_WNC_GH.yaml:/opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_P5G_WNC_GH.yaml + - ./aerial_l1_entrypoint.sh:/opt/nvidia/cuBB/aerial_l1_entrypoint.sh + - /var/log/aerial:/var/log/aerial + - ../../../cmake_targets/share:/opt/cuBB/share +``` + +By applying this change, it would easier for the user to modify the L1 code or +configuration files directly from the `aerial-cuda-accelerated-ran` repository. + +`ci-scripts/yaml_files/sa_gnb_aerial/docker-compose.yaml` is tailored for our +CI. +We copy the `aerial-cuda-accelerated-ran` repository inside the cuBB +docker image using `docker cp` and commit the changes using `docker commit`. +We mount the `cuphycontroller_P5G_WNC_GH.yaml` in the `nv-cubb` volumes so +that it would override some parameters depending on the setup being run on CI. + +After building the gNB image, and preparing the configuration file, the setup +can be run with the following command: ```bash cd ci-scripts/yaml_files/sa_gnb_aerial/ docker compose up -d - ``` -This will start both containers, beginning with `nv-cubb`, and `oai-gnb-aerial` will start only after it is ready. +This will start both containers, beginning with `nv-cubb`, and +`oai-gnb-aerial` will start only after it is ready. The logs can be followed using these commands: @@ -263,8 +343,13 @@ The logs can be followed using these commands: docker logs -f oai-gnb-aerial docker logs -f nv-cubb ``` + #### Running with multiple L2s -One L1 instance can support multiple L2 instances. See also the [aerial documentation](https://developer.nvidia.com/docs/gputelecom/aerial-sdk/text/cubb_quickstart/running_cubb-end-to-end.html#run-multiple-l2-instances-with-single-l1-instance) for more details. + +One L1 instance can support multiple L2 instances. See also +the [aerial +documentation](https://docs.nvidia.com/aerial/cuda-accelerated-ran/latest/quickstart_guide/running_cubb-end-to-end.html#run-multiple-l2-instances-with-single-l1-instance) +for more details. In OAI the shared memory prefix must be configured in the configuration file. @@ -274,11 +359,12 @@ In OAI the shared memory prefix must be configured in the configuration file. ``` -### Stopping the setup +#### Stopping the setup + +Run the following command to stop and remove both containers, leaving the +system ready to be restarted later: -Run the following command to stop and remove both containers, leaving the system ready to be restarted later: ```bash cd ci-scripts/yaml_files/sa_gnb_aerial/ docker compose down - ```