From e615321eb2dd1176d83572150152cd8a029e352e Mon Sep 17 00:00:00 2001 From: Robert Schmidt Date: Thu, 7 Mar 2024 11:42:41 +0100 Subject: [PATCH] Rewrite debugging instructions for CI fail --- doc/TESTBenches.md | 91 +++++++++++++++++++++++++++++----------------- 1 file changed, 58 insertions(+), 33 deletions(-) diff --git a/doc/TESTBenches.md b/doc/TESTBenches.md index fa488a42f7..0b5b72679c 100644 --- a/doc/TESTBenches.md +++ b/doc/TESTBenches.md @@ -250,7 +250,44 @@ Some tests are run from source (e.g. `ci-scripts/xml_files/gnb_phytest_usrp_run.xml`), which directly give the options they are run with. -## How to retrieve core dumps (for CI team members) +## How to debug CI failures + +It is possible to debug CI failures using the generated core dump and the image +used for the run. A script is provided (see developer instructions below) that, +provided the core dump file, container image, and the source tree, executes +`gdb` inside the container; using the core dump information, a developer can +investigate the cause of failure. + +### Developer instructions + +The CI team will send you a docker image and a core dump file, and the commit +as of which the pipeline failed. Let's assume the coredump is stored at +`/tmp/coredump.tar.xz`, and the image is in `/tmp/oai-nr-ue.tar.gz`. First, you +should check out the corresponding branch (or directly the commit), let's say +in `~/oai-branch-fail`. Now, unpack the core dump, load the image into docker, +and use the script [`docker/debug_core_image.sh`](../docker/debug_core_image.sh) +to open gdb, as follows: + +``` +cd /tmp +tar -xJf /tmp/coredump.tar.xz +docker load < /tmp/oai-nr-ue.tar.gz +~/oai-branch-fail/docker/debug_core_image.sh /tmp/coredump ~/oai-branch-fail +``` + +where you replace `` with the image loaded in `docker load`. The script +will start the container and open gdb; you should see information about where +the failure (e.g., segmentation fault) happened. If you just see `??`, the core +dump and container image don't match. Be also on the lookout for the +corresponding message from gdb: +``` +warning: core file may not match specified executable file. +``` + +Once you quit `gdb`, the container image will be removed automatically. + + +### CI team instructions The entrypoint scripts of all containers print the core pattern that is used on the running machine. Search for `core_pattern` at the start of the container @@ -262,19 +299,14 @@ logs to retrieve the possible location. Possible locations might be: - abrt: see [documentation](https://abrt.readthedocs.io/en/latest/usage.html) - apport: see [documentation](https://wiki.ubuntu.com/Apport) -You furthermore have to extract the executable that caused the core dump. -Download the container image, and extract, e.g.: +See below for instructions on how to retrieve the core dump. Further, download +the image and store it to a file using `docker save`. Make sure to pick the +right image (Ubuntu or RHEL)! -``` -docker create --name c1 porcepix.sboai.cs.eurecom.fr/oai-gnb:develop-c99db698 -docker cp c1:/opt/oai-gnb/bin/nr-softmodem /tmp -docker rm c1 -``` - -### Core dump in a file +#### Core dump in a file **This is not recommended, as files could pile up and fill the system disk -completely!** Prefer systemd or abrt instead. +completely!** Prefer another method further down. If the core pattern is a path: it should at least include the time in the pattern name (suggested pattern: `/tmp/core.%e.%p.%t`) to correlate the time @@ -282,38 +314,31 @@ the segfault occurred with the CI logs. If you identified the core dump, copy the core dump from that machine; if identification is difficult, consider rerunning the pipeline. -### Core dump via systemd +#### Core dump via systemd -Run this command to list all core dumps: +Use the first command to list all core dumps. Scroll down to the core dump of +interest (it lists the executables in the last column; use the time to +correlate the segfault and the CI run). Take the PID of the executable (first +column after the time). Dump the core dump to a location of your choice. ``` sudo coredumpctl list -``` - -Scroll to the end and find the core dump of interest (it lists the executables -in the last column; use the time to correlate the segfault and the CI run). -Take the PID of the executable (first column after the time). Dump the core -dump to a location of your choice: - -``` sudo coredumpctl dump > /tmp/coredump ``` -### Core dump via abrt (automatic bug reporting tool) +#### Core dump via abrt (automatic bug reporting tool) TBD: use the documentation page for the moment. -### Core dump via apport - -On Ubuntu machines, apport first needs to be enabled to collect core dumps: +#### Core dump via apport +I did not find an easy way to use apport. Anyway, the systemd approach works +fine. So remove apport, install systemd-coredump, and verify it is the new +coredump handler: ``` -sudo systemctl enable apport.service -``` - -and [needs to be enabled](https://wiki.ubuntu.com/Apport#How_to_enable_apport). -Then, show a list of core dumps using - -``` -sudo apport-cli +sudo systemctl stop apport +sudo systemctl mask --now apport +sudo apt install systemd-coredump +# Verify this changed the core pattern to a pipe to systemd-coredump +sysctl kernel.core_pattern ```