From 13517cd6bc130373619eab06382089e25e0a97c4 Mon Sep 17 00:00:00 2001 From: Robert Schmidt Date: Thu, 14 Mar 2024 18:21:03 +0100 Subject: [PATCH] Write documentation on performance tuning and security --- doc/NR_SA_Tutorial_COTS_UE.md | 2 + doc/README.md | 1 + doc/tuning_and_security.md | 165 ++++++++++++++++++++++++++++++++++ radio/rfsimulator/README.md | 2 + 4 files changed, 170 insertions(+) create mode 100644 doc/tuning_and_security.md diff --git a/doc/NR_SA_Tutorial_COTS_UE.md b/doc/NR_SA_Tutorial_COTS_UE.md index 0924eee1b0..291dfdd7b4 100644 --- a/doc/NR_SA_Tutorial_COTS_UE.md +++ b/doc/NR_SA_Tutorial_COTS_UE.md @@ -163,6 +163,8 @@ docker exec -it oai-ext-dn iperf -u -t 86400 -i 1 -fk -B 192.168.70.135 -b 100M # 6. Advanced configurations (optional) +See also the [dedicated document on performance tuning](./tuning_and_security.md). + ## 6.1 USRP N300 and X300 Ethernet Tuning Please also refer to the official [USRP Host Performance Tuning Tips and Tricks](https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks) tuning guide. diff --git a/doc/README.md b/doc/README.md index ef72249725..1c4413a8a5 100644 --- a/doc/README.md +++ b/doc/README.md @@ -24,6 +24,7 @@ - [clang-format.md](./clang-format.md): how to format the code - [sanitizers.md](./dev_tools/sanitizers.md): how to run with ASan/UBSan/MemSAN/TSan - [environment-variables.md](./environment-variables.md): the environment variables used by OAI +- [tuning_and_security.md](./tuning_and_security.md): performance and security considerations There is some general information in the [OpenAirInterface Gitlab Wiki](https://gitlab.eurecom.fr/oai/openairinterface5g/-/wikis/home) diff --git a/doc/tuning_and_security.md b/doc/tuning_and_security.md new file mode 100644 index 0000000000..94b6ceb07c --- /dev/null +++ b/doc/tuning_and_security.md @@ -0,0 +1,165 @@ +This document outlines some information specific to system tuning for +running all executables (`nr-softmodem`, `lte-softmodem`, `nr-uesoftmodem`, +`lte-uesoftmodem`, `nr-cuup`, called softmodems for short in the following). +Also, it explains which Linux capabilities are required to run, and how to run +without sudo. + +[TOC] + +# Performance Tuning + +Please also refer to the [advanced configuration in the +tutorial](NR_SA_Tutorial_COTS_UE.md#6-advanced-configurations-optional), which +groups many tips and tricks. + +## CPU + +OAI used to try to set the minimum CPU-DMA latency to 2 us by writing to +`/dev/cpu_dma_latency`. However, it is unclear if this has a significant +effect. Further, in containerized workloads, it might not be possible to set +this at all. We assume the user of OAI sets this value before starting OAI. +See the [Linux kernel +documentation](https://www.kernel.org/doc/html/latest/admin-guide/pm/cpuidle.html) +for more information. + +OAI used to try to set the minimum frequency of cores running OAI to the +maximum frequency. We assume that the user sets CPU frequency policies +accordingly; in fact, tutorials generally suggest to either set a performance +CPU governor, or disable any sleep states at all. Hence, setting these +low-level parameters seems useless, and we assume the user of OAI disables +sleep states before starting OAI. + +To disable all sleep states, simply run the following: +``` +sudo cpupower idle-set -D0 +``` +Sometimes, dependencies might not be installed, in which case you should +install what `cpupower` asks you to install. Note that this is not persistent. + +See the [Linux kernel +documentation](https://www.kernel.org/doc/html/latest/admin-guide/pm/cpufreq.html) +for more information. + +You can disable hyper-threading in the BIOS. + +You can disable KPTI Protections for Spectre/Meltdown for more performance. +**This is a security risk.** Add `mitigations=off nosmt` in your grub config and +update grub. + +## Ethernet-based Radios + +For ethernet-based radios, such as AW2S, some USRPs, and 7.2 radios, increase +the ringbuffers to a maximum. Read on interface `` +using option `-g`, then set it with `-G`: + +``` +ethtool -g +ethtool -G rx tx +``` + +Also, you can increase the kernel's default and maximum read and write socket +buffer sizes to a high values, e.g., 134217728: + +``` +sudo sysctl -n -e -q -w net.core.rmem_default=134217728 +sudo sysctl -n -e -q -w net.core.rmem_max=134217728 +sudo sysctl -n -e -q -w net.core.wmem_default=134217728 +sudo sysctl -n -e -q -w net.core.wmem_max=134217728 +``` + +# Capabilities + +Historically, all softmodems are executed as `root`, typically using `sudo`. +This remains a possibility, but we do not recommend it due to security +considerations. Rather, consider giving specific capabilities as outlined +below. Read `capabilities(7)` (`man 7 capabilities`) for more information on +each of the below capabilities. + +Note that we tested this using 5G executables; 4G should work, but have not +been tested as extensively. If in doubt, run eNB/lteUE using `sudo`. The below +comments on capabilities apply in general as well; however, 4G executable might +not warn about missing capabilities or just fail. + +Refer to any of the docker-compose files under `ci-scripts/` to see how to give +capabilities in docker. If you run from source, you can use `setcap` to mark a +process to run with specific capabilities. For instance, you can add the "general +capabilities" as described further below on the files like this: + +``` +sudo setcap cap_sys_nice+ep ./nr-softmodem +sudo setcap cap_ipc_lock+ep ./nr-softmodem +sudo setcap cap_sys_nice+ep ./nr-uesoftmodem +sudo setcap cap_ipc_lock+ep ./nr-uesoftmodem +sudo setcap cap_net_admin+ep ./nr-uesoftmodem +``` + +To make only temporary changes to capabilities, use `capsh`. It needs to be +started with all capabilities, so needs `sudo`. To drop all capabilities, +issue: +``` +sudo -E capsh --drop="cap_sys_nice,cap_chown,cap_dac_read_search,cap_fowner,cap_fsetid,\ +cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,\ +cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,\ +cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,\ +cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,\ +cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,\ +cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore" --print -- \ + -c "/absolute/path/to/nr-softmodem -O /absolute/path/to/config.conf" +``` + +(For readability, the command has been separated onto multiple lines through +`\`). To run with `SYS_NICE`, remove the first capability (`cap_sys_nice`) +from the list of dropped capabilities. + +## General capabilities + +- `SYS_NICE`: required by all softmodems to assign a higher priority to + threads to increase the likelihood that the Linux scheduler schedules a + thread for CPU time. Also, in some configurations, CPU pinning is employed to + ensure consistent performance, also known as setting a CPU affinity. + + This capability is necessary by default when running any softmodem, for + setting real-time requirements on processing threads. To allow to run without + these requirements, the softmodems check if `SYS_NICE` is available, and + skips any thread priority and affinity settings if the capability is not + available. This allows to run any softmodem without root privileges in RFsim; + you can see this by either observing a corresponding warning at the beginning + of the execution, or the fact that no affinity/default priority is set for + new threads. + +- `IPC_LOCK`: OAI tries to lock all its virtual address space into RAM, + preventing that memory from being paged to the swap area. If this capability + is not present, a warning is printed. + +- `NET_ADMIN`: Required at the UE to open the interface `oaitun_ue1` or eNB/gNB + in noS1 mode. 5G executables will throw an error if they cannot create or + modify an interface, but will continue running. It is therefore possible to + run without this capability, but you cannot inject any traffic into the + system. 4G executables might need this requirement, and possibly fail. + +## Capabilities with DPDK + +Additionally to the "general capabilities" above, you need these capabilities +when running with 7.2 fronthaul, which uses the xran library with a dependency +on DPDK: + +- `IPC_LOCK` (becomes mandatory with DPDK) +- `SYS_RESOURCE` +- `NET_RAW` + +## Capabilities with UHD + +You don't need any additional capabilities for UHD beyond the "general +capabilities" for performance outlined above. Make sure that the USB device(s) +are readable and writable by the current user (e.g., `uhd_usrp_probe` can be +executed by a non-root user). + +## Capabilities with AW2S + +You don't need any additional capabilities for AW2S beyond the "general +capabilities" for performance outlined above. + +## Other radios + +Other radios have not been tested. If they do not work without additional +capabilities beyond the "general capabilities", please file a bug report. diff --git a/radio/rfsimulator/README.md b/radio/rfsimulator/README.md index ea2188d4df..ba63ae4ec0 100644 --- a/radio/rfsimulator/README.md +++ b/radio/rfsimulator/README.md @@ -189,3 +189,5 @@ to reduce the amount of transported samples: - Prefer smaller cell bandwidths A possible, unimplemented optimization would be to compress samples. + +You can further [tune your machine](../../doc/tuning_and_security.md)