Use timeout command to kill hangling iperf3

In some pipelines, it happens that iperf3 is hanging on 100% CPU load.
In these cases, a simple TERM signal does not seem to be enough to kill
it; KILL is necessary.

While e.g., RemoteCmd's driver paramiko has a timeout command, the
documentation [1] is not really clear how it is trying to stop commands.
Since it effectively does not stop, we have to assume it is only sending
TERM.

To remedy this, use the timeout command to start iperf3. We use option
-v for diagnose timeouts on stderr, and send KILL after an additional 3
seconds (-k3) if TERM does not succeed. The timeout is sent to the same
duration as the overall iperf timeout.

[1] https://docs.paramiko.org/en/3.4/api/channel.html#paramiko.channel.Channel.settimeout
This commit is contained in:
Robert Schmidt
2024-05-24 14:20:04 +02:00
parent 9249131046
commit 8d16ff8459
+3 -2
View File
@@ -818,10 +818,11 @@ class OaiCiTest():
with cls_cmd.getConnection(ue.getHost()) as cmd_ue, cls_cmd.getConnection(EPC.IPAddress) as cmd_svr:
port = 5002 + idx
# note: some core setups start an iperf3 server automatically, indicated in ci_infra by runIperf3Server: False`
t = iperf_time * 1.5
if runIperf3Server:
cmd_svr.run(f'{svr.getCmdPrefix()} nohup iperf3 -s -B {svrIP} -p {port} -1 {jsonReport} &', timeout=iperf_time*1.5)
cmd_svr.run(f'{svr.getCmdPrefix()} nohup timeout -vk3 {t} iperf3 -s -B {svrIP} -p {port} -1 {jsonReport} &', timeout=t)
cmd_ue.run(f'rm /tmp/{client_filename}', reportNonZero=False)
cmd_ue.run(f'{ue.getCmdPrefix()} {iperf_ue} -B {ueIP} -c {svrIP} -p {port} {iperf_opt} {jsonReport} {serverReport} -O 5 >> /tmp/{client_filename}', timeout=iperf_time*1.5)
cmd_ue.run(f'{ue.getCmdPrefix()} timeout -vk3 {t} {iperf_ue} -B {ueIP} -c {svrIP} -p {port} {iperf_opt} {jsonReport} {serverReport} -O 5 >> /tmp/{client_filename}', timeout=t)
if svr.getHost() == 'localhost':
cmd_ue.run(f'mkdir -p {logPath}')
cmd_ue.run(f'cp /tmp/{client_filename} {logPath}/{client_filename}')