Write documentation on performance tuning and security
This commit is contained in:
Robert Schmidt
@@ -163,6 +163,8 @@ docker exec -it oai-ext-dn iperf -u -t 86400 -i 1 -fk -B 192.168.70.135 -b 100M
|
||||
|
||||
# 6. Advanced configurations (optional)
|
||||
|
||||
See also the [dedicated document on performance tuning](./tuning_and_security.md).
|
||||
|
||||
## 6.1 USRP N300 and X300 Ethernet Tuning
|
||||
|
||||
Please also refer to the official [USRP Host Performance Tuning Tips and Tricks](https://kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks) tuning guide.
|
||||
|
||||
@@ -24,6 +24,7 @@
|
||||
- [clang-format.md](./clang-format.md): how to format the code
|
||||
- [sanitizers.md](./dev_tools/sanitizers.md): how to run with ASan/UBSan/MemSAN/TSan
|
||||
- [environment-variables.md](./environment-variables.md): the environment variables used by OAI
|
||||
- [tuning_and_security.md](./tuning_and_security.md): performance and security considerations
|
||||
|
||||
There is some general information in the [OpenAirInterface Gitlab Wiki](https://gitlab.eurecom.fr/oai/openairinterface5g/-/wikis/home)
|
||||
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
This document outlines some information specific to system tuning for
|
||||
running all executables (`nr-softmodem`, `lte-softmodem`, `nr-uesoftmodem`,
|
||||
`lte-uesoftmodem`, `nr-cuup`, called softmodems for short in the following).
|
||||
Also, it explains which Linux capabilities are required to run, and how to run
|
||||
without sudo.
|
||||
|
||||
[TOC]
|
||||
|
||||
# Performance Tuning
|
||||
|
||||
Please also refer to the [advanced configuration in the
|
||||
tutorial](NR_SA_Tutorial_COTS_UE.md#6-advanced-configurations-optional), which
|
||||
groups many tips and tricks.
|
||||
|
||||
## CPU
|
||||
|
||||
OAI used to try to set the minimum CPU-DMA latency to 2 us by writing to
|
||||
`/dev/cpu_dma_latency`. However, it is unclear if this has a significant
|
||||
effect. Further, in containerized workloads, it might not be possible to set
|
||||
this at all. We assume the user of OAI sets this value before starting OAI.
|
||||
See the [Linux kernel
|
||||
documentation](https://www.kernel.org/doc/html/latest/admin-guide/pm/cpuidle.html)
|
||||
for more information.
|
||||
|
||||
OAI used to try to set the minimum frequency of cores running OAI to the
|
||||
maximum frequency. We assume that the user sets CPU frequency policies
|
||||
accordingly; in fact, tutorials generally suggest to either set a performance
|
||||
CPU governor, or disable any sleep states at all. Hence, setting these
|
||||
low-level parameters seems useless, and we assume the user of OAI disables
|
||||
sleep states before starting OAI.
|
||||
|
||||
To disable all sleep states, simply run the following:
|
||||
```
|
||||
sudo cpupower idle-set -D0
|
||||
```
|
||||
Sometimes, dependencies might not be installed, in which case you should
|
||||
install what `cpupower` asks you to install. Note that this is not persistent.
|
||||
|
||||
See the [Linux kernel
|
||||
documentation](https://www.kernel.org/doc/html/latest/admin-guide/pm/cpufreq.html)
|
||||
for more information.
|
||||
|
||||
You can disable hyper-threading in the BIOS.
|
||||
|
||||
You can disable KPTI Protections for Spectre/Meltdown for more performance.
|
||||
**This is a security risk.** Add `mitigations=off nosmt` in your grub config and
|
||||
update grub.
|
||||
|
||||
## Ethernet-based Radios
|
||||
|
||||
For ethernet-based radios, such as AW2S, some USRPs, and 7.2 radios, increase
|
||||
the ringbuffers to a maximum. Read on interface `<fronthaul-interface-name>`
|
||||
using option `-g`, then set it with `-G`:
|
||||
|
||||
```
|
||||
ethtool -g <fronthaul-interface-name>
|
||||
ethtool -G <fronthaul-interface-name> rx <maximum-rx-value> tx <maixmum-tx-value>
|
||||
```
|
||||
|
||||
Also, you can increase the kernel's default and maximum read and write socket
|
||||
buffer sizes to a high values, e.g., 134217728:
|
||||
|
||||
```
|
||||
sudo sysctl -n -e -q -w net.core.rmem_default=134217728
|
||||
sudo sysctl -n -e -q -w net.core.rmem_max=134217728
|
||||
sudo sysctl -n -e -q -w net.core.wmem_default=134217728
|
||||
sudo sysctl -n -e -q -w net.core.wmem_max=134217728
|
||||
```
|
||||
|
||||
# Capabilities
|
||||
|
||||
Historically, all softmodems are executed as `root`, typically using `sudo`.
|
||||
This remains a possibility, but we do not recommend it due to security
|
||||
considerations. Rather, consider giving specific capabilities as outlined
|
||||
below. Read `capabilities(7)` (`man 7 capabilities`) for more information on
|
||||
each of the below capabilities.
|
||||
|
||||
Note that we tested this using 5G executables; 4G should work, but have not
|
||||
been tested as extensively. If in doubt, run eNB/lteUE using `sudo`. The below
|
||||
comments on capabilities apply in general as well; however, 4G executable might
|
||||
not warn about missing capabilities or just fail.
|
||||
|
||||
Refer to any of the docker-compose files under `ci-scripts/` to see how to give
|
||||
capabilities in docker. If you run from source, you can use `setcap` to mark a
|
||||
process to run with specific capabilities. For instance, you can add the "general
|
||||
capabilities" as described further below on the files like this:
|
||||
|
||||
```
|
||||
sudo setcap cap_sys_nice+ep ./nr-softmodem
|
||||
sudo setcap cap_ipc_lock+ep ./nr-softmodem
|
||||
sudo setcap cap_sys_nice+ep ./nr-uesoftmodem
|
||||
sudo setcap cap_ipc_lock+ep ./nr-uesoftmodem
|
||||
sudo setcap cap_net_admin+ep ./nr-uesoftmodem
|
||||
```
|
||||
|
||||
To make only temporary changes to capabilities, use `capsh`. It needs to be
|
||||
started with all capabilities, so needs `sudo`. To drop all capabilities,
|
||||
issue:
|
||||
```
|
||||
sudo -E capsh --drop="cap_sys_nice,cap_chown,cap_dac_read_search,cap_fowner,cap_fsetid,\
|
||||
cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,\
|
||||
cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,\
|
||||
cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,\
|
||||
cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,\
|
||||
cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,\
|
||||
cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore" --print -- \
|
||||
-c "/absolute/path/to/nr-softmodem -O /absolute/path/to/config.conf"
|
||||
```
|
||||
|
||||
(For readability, the command has been separated onto multiple lines through
|
||||
`\`). To run with `SYS_NICE`, remove the first capability (`cap_sys_nice`)
|
||||
from the list of dropped capabilities.
|
||||
|
||||
## General capabilities
|
||||
|
||||
- `SYS_NICE`: required by all softmodems to assign a higher priority to
|
||||
threads to increase the likelihood that the Linux scheduler schedules a
|
||||
thread for CPU time. Also, in some configurations, CPU pinning is employed to
|
||||
ensure consistent performance, also known as setting a CPU affinity.
|
||||
|
||||
This capability is necessary by default when running any softmodem, for
|
||||
setting real-time requirements on processing threads. To allow to run without
|
||||
these requirements, the softmodems check if `SYS_NICE` is available, and
|
||||
skips any thread priority and affinity settings if the capability is not
|
||||
available. This allows to run any softmodem without root privileges in RFsim;
|
||||
you can see this by either observing a corresponding warning at the beginning
|
||||
of the execution, or the fact that no affinity/default priority is set for
|
||||
new threads.
|
||||
|
||||
- `IPC_LOCK`: OAI tries to lock all its virtual address space into RAM,
|
||||
preventing that memory from being paged to the swap area. If this capability
|
||||
is not present, a warning is printed.
|
||||
|
||||
- `NET_ADMIN`: Required at the UE to open the interface `oaitun_ue1` or eNB/gNB
|
||||
in noS1 mode. 5G executables will throw an error if they cannot create or
|
||||
modify an interface, but will continue running. It is therefore possible to
|
||||
run without this capability, but you cannot inject any traffic into the
|
||||
system. 4G executables might need this requirement, and possibly fail.
|
||||
|
||||
## Capabilities with DPDK
|
||||
|
||||
Additionally to the "general capabilities" above, you need these capabilities
|
||||
when running with 7.2 fronthaul, which uses the xran library with a dependency
|
||||
on DPDK:
|
||||
|
||||
- `IPC_LOCK` (becomes mandatory with DPDK)
|
||||
- `SYS_RESOURCE`
|
||||
- `NET_RAW`
|
||||
|
||||
## Capabilities with UHD
|
||||
|
||||
You don't need any additional capabilities for UHD beyond the "general
|
||||
capabilities" for performance outlined above. Make sure that the USB device(s)
|
||||
are readable and writable by the current user (e.g., `uhd_usrp_probe` can be
|
||||
executed by a non-root user).
|
||||
|
||||
## Capabilities with AW2S
|
||||
|
||||
You don't need any additional capabilities for AW2S beyond the "general
|
||||
capabilities" for performance outlined above.
|
||||
|
||||
## Other radios
|
||||
|
||||
Other radios have not been tested. If they do not work without additional
|
||||
capabilities beyond the "general capabilities", please file a bug report.
|
||||
Reference in New Issue
Block a user