166 lines
7.0 KiB
Markdown
166 lines
7.0 KiB
Markdown
This document outlines some information specific to system tuning for
|
|
running all executables (`nr-softmodem`, `lte-softmodem`, `nr-uesoftmodem`,
|
|
`lte-uesoftmodem`, `nr-cuup`, called softmodems for short in the following).
|
|
Also, it explains which Linux capabilities are required to run, and how to run
|
|
without sudo.
|
|
|
|
[TOC]
|
|
|
|
# Performance Tuning
|
|
|
|
Please also refer to the [advanced configuration in the
|
|
tutorial](NR_SA_Tutorial_COTS_UE.md#6-advanced-configurations-optional), which
|
|
groups many tips and tricks.
|
|
|
|
## CPU
|
|
|
|
OAI used to try to set the minimum CPU-DMA latency to 2 us by writing to
|
|
`/dev/cpu_dma_latency`. However, it is unclear if this has a significant
|
|
effect. Further, in containerized workloads, it might not be possible to set
|
|
this at all. We assume the user of OAI sets this value before starting OAI.
|
|
See the [Linux kernel
|
|
documentation](https://www.kernel.org/doc/html/latest/admin-guide/pm/cpuidle.html)
|
|
for more information.
|
|
|
|
OAI used to try to set the minimum frequency of cores running OAI to the
|
|
maximum frequency. We assume that the user sets CPU frequency policies
|
|
accordingly; in fact, tutorials generally suggest to either set a performance
|
|
CPU governor, or disable any sleep states at all. Hence, setting these
|
|
low-level parameters seems useless, and we assume the user of OAI disables
|
|
sleep states before starting OAI.
|
|
|
|
To disable all sleep states, simply run the following:
|
|
```
|
|
sudo cpupower idle-set -D0
|
|
```
|
|
Sometimes, dependencies might not be installed, in which case you should
|
|
install what `cpupower` asks you to install. Note that this is not persistent.
|
|
|
|
See the [Linux kernel
|
|
documentation](https://www.kernel.org/doc/html/latest/admin-guide/pm/cpufreq.html)
|
|
for more information.
|
|
|
|
You can disable hyper-threading in the BIOS.
|
|
|
|
You can disable KPTI Protections for Spectre/Meltdown for more performance.
|
|
**This is a security risk.** Add `mitigations=off nosmt` in your grub config and
|
|
update grub.
|
|
|
|
## Ethernet-based Radios
|
|
|
|
For ethernet-based radios, such as AW2S, some USRPs, and 7.2 radios, increase
|
|
the ringbuffers to a maximum. Read on interface `<fronthaul-interface-name>`
|
|
using option `-g`, then set it with `-G`:
|
|
|
|
```
|
|
ethtool -g <fronthaul-interface-name>
|
|
ethtool -G <fronthaul-interface-name> rx <maximum-rx-value> tx <maixmum-tx-value>
|
|
```
|
|
|
|
Also, you can increase the kernel's default and maximum read and write socket
|
|
buffer sizes to a high values, e.g., 134217728:
|
|
|
|
```
|
|
sudo sysctl -n -e -q -w net.core.rmem_default=134217728
|
|
sudo sysctl -n -e -q -w net.core.rmem_max=134217728
|
|
sudo sysctl -n -e -q -w net.core.wmem_default=134217728
|
|
sudo sysctl -n -e -q -w net.core.wmem_max=134217728
|
|
```
|
|
|
|
# Capabilities
|
|
|
|
Historically, all softmodems are executed as `root`, typically using `sudo`.
|
|
This remains a possibility, but we do not recommend it due to security
|
|
considerations. Rather, consider giving specific capabilities as outlined
|
|
below. Read `capabilities(7)` (`man 7 capabilities`) for more information on
|
|
each of the below capabilities.
|
|
|
|
Note that we tested this using 5G executables; 4G should work, but have not
|
|
been tested as extensively. If in doubt, run eNB/lteUE using `sudo`. The below
|
|
comments on capabilities apply in general as well; however, 4G executable might
|
|
not warn about missing capabilities or just fail.
|
|
|
|
Refer to any of the docker-compose files under `ci-scripts/` to see how to give
|
|
capabilities in docker. If you run from source, you can use `setcap` to mark a
|
|
process to run with specific capabilities. For instance, you can add the "general
|
|
capabilities" as described further below on the files like this:
|
|
|
|
```
|
|
sudo setcap cap_sys_nice+ep ./nr-softmodem
|
|
sudo setcap cap_ipc_lock+ep ./nr-softmodem
|
|
sudo setcap cap_sys_nice+ep ./nr-uesoftmodem
|
|
sudo setcap cap_ipc_lock+ep ./nr-uesoftmodem
|
|
sudo setcap cap_net_admin+ep ./nr-uesoftmodem
|
|
```
|
|
|
|
To make only temporary changes to capabilities, use `capsh`. It needs to be
|
|
started with all capabilities, so needs `sudo`. To drop all capabilities,
|
|
issue:
|
|
```
|
|
sudo -E capsh --drop="cap_sys_nice,cap_chown,cap_dac_read_search,cap_fowner,cap_fsetid,\
|
|
cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,\
|
|
cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,\
|
|
cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,\
|
|
cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,\
|
|
cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,\
|
|
cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore" --print -- \
|
|
-c "/absolute/path/to/nr-softmodem -O /absolute/path/to/config.conf"
|
|
```
|
|
|
|
(For readability, the command has been separated onto multiple lines through
|
|
`\`). To run with `SYS_NICE`, remove the first capability (`cap_sys_nice`)
|
|
from the list of dropped capabilities.
|
|
|
|
## General capabilities
|
|
|
|
- `SYS_NICE`: required by all softmodems to assign a higher priority to
|
|
threads to increase the likelihood that the Linux scheduler schedules a
|
|
thread for CPU time. Also, in some configurations, CPU pinning is employed to
|
|
ensure consistent performance, also known as setting a CPU affinity.
|
|
|
|
This capability is necessary by default when running any softmodem, for
|
|
setting real-time requirements on processing threads. To allow to run without
|
|
these requirements, the softmodems check if `SYS_NICE` is available, and
|
|
skips any thread priority and affinity settings if the capability is not
|
|
available. This allows to run any softmodem without root privileges in RFsim;
|
|
you can see this by either observing a corresponding warning at the beginning
|
|
of the execution, or the fact that no affinity/default priority is set for
|
|
new threads.
|
|
|
|
- `IPC_LOCK`: OAI tries to lock all its virtual address space into RAM,
|
|
preventing that memory from being paged to the swap area. If this capability
|
|
is not present, a warning is printed.
|
|
|
|
- `NET_ADMIN`: Required at the UE to open the interface `oaitun_ue1` or eNB/gNB
|
|
in noS1 mode. 5G executables will throw an error if they cannot create or
|
|
modify an interface, but will continue running. It is therefore possible to
|
|
run without this capability, but you cannot inject any traffic into the
|
|
system. 4G executables might need this requirement, and possibly fail.
|
|
|
|
## Capabilities with DPDK
|
|
|
|
Additionally to the "general capabilities" above, you need these capabilities
|
|
when running with 7.2 fronthaul, which uses the xran library with a dependency
|
|
on DPDK:
|
|
|
|
- `IPC_LOCK` (becomes mandatory with DPDK)
|
|
- `SYS_RESOURCE`
|
|
- `NET_RAW`
|
|
|
|
## Capabilities with UHD
|
|
|
|
You don't need any additional capabilities for UHD beyond the "general
|
|
capabilities" for performance outlined above. Make sure that the USB device(s)
|
|
are readable and writable by the current user (e.g., `uhd_usrp_probe` can be
|
|
executed by a non-root user).
|
|
|
|
## Capabilities with AW2S
|
|
|
|
You don't need any additional capabilities for AW2S beyond the "general
|
|
capabilities" for performance outlined above.
|
|
|
|
## Other radios
|
|
|
|
Other radios have not been tested. If they do not work without additional
|
|
capabilities beyond the "general capabilities", please file a bug report.
|