diff --git a/doc/TESTBenches.md b/doc/TESTBenches.md index 36639371b3..433e696e79 100644 --- a/doc/TESTBenches.md +++ b/doc/TESTBenches.md @@ -255,7 +255,44 @@ Some tests are run from source (e.g. `ci-scripts/xml_files/gnb_phytest_usrp_run.xml`), which directly give the options they are run with. -## How to retrieve core dumps (for CI team members) +## How to debug CI failures + +It is possible to debug CI failures using the generated core dump and the image +used for the run. A script is provided (see developer instructions below) that, +provided the core dump file, container image, and the source tree, executes +`gdb` inside the container; using the core dump information, a developer can +investigate the cause of failure. + +### Developer instructions + +The CI team will send you a docker image and a core dump file, and the commit +as of which the pipeline failed. Let's assume the coredump is stored at +`/tmp/coredump.tar.xz`, and the image is in `/tmp/oai-nr-ue.tar.gz`. First, you +should check out the corresponding branch (or directly the commit), let's say +in `~/oai-branch-fail`. Now, unpack the core dump, load the image into docker, +and use the script [`docker/debug_core_image.sh`](../docker/debug_core_image.sh) +to open gdb, as follows: + +``` +cd /tmp +tar -xJf /tmp/coredump.tar.xz +docker load < /tmp/oai-nr-ue.tar.gz +~/oai-branch-fail/docker/debug_core_image.sh /tmp/coredump ~/oai-branch-fail +``` + +where you replace `` with the image loaded in `docker load`. The script +will start the container and open gdb; you should see information about where +the failure (e.g., segmentation fault) happened. If you just see `??`, the core +dump and container image don't match. Be also on the lookout for the +corresponding message from gdb: +``` +warning: core file may not match specified executable file. +``` + +Once you quit `gdb`, the container image will be removed automatically. + + +### CI team instructions The entrypoint scripts of all containers print the core pattern that is used on the running machine. Search for `core_pattern` at the start of the container @@ -267,19 +304,14 @@ logs to retrieve the possible location. Possible locations might be: - abrt: see [documentation](https://abrt.readthedocs.io/en/latest/usage.html) - apport: see [documentation](https://wiki.ubuntu.com/Apport) -You furthermore have to extract the executable that caused the core dump. -Download the container image, and extract, e.g.: +See below for instructions on how to retrieve the core dump. Further, download +the image and store it to a file using `docker save`. Make sure to pick the +right image (Ubuntu or RHEL)! -``` -docker create --name c1 porcepix.sboai.cs.eurecom.fr/oai-gnb:develop-c99db698 -docker cp c1:/opt/oai-gnb/bin/nr-softmodem /tmp -docker rm c1 -``` - -### Core dump in a file +#### Core dump in a file **This is not recommended, as files could pile up and fill the system disk -completely!** Prefer systemd or abrt instead. +completely!** Prefer another method further down. If the core pattern is a path: it should at least include the time in the pattern name (suggested pattern: `/tmp/core.%e.%p.%t`) to correlate the time @@ -287,38 +319,31 @@ the segfault occurred with the CI logs. If you identified the core dump, copy the core dump from that machine; if identification is difficult, consider rerunning the pipeline. -### Core dump via systemd +#### Core dump via systemd -Run this command to list all core dumps: +Use the first command to list all core dumps. Scroll down to the core dump of +interest (it lists the executables in the last column; use the time to +correlate the segfault and the CI run). Take the PID of the executable (first +column after the time). Dump the core dump to a location of your choice. ``` sudo coredumpctl list -``` - -Scroll to the end and find the core dump of interest (it lists the executables -in the last column; use the time to correlate the segfault and the CI run). -Take the PID of the executable (first column after the time). Dump the core -dump to a location of your choice: - -``` sudo coredumpctl dump > /tmp/coredump ``` -### Core dump via abrt (automatic bug reporting tool) +#### Core dump via abrt (automatic bug reporting tool) TBD: use the documentation page for the moment. -### Core dump via apport - -On Ubuntu machines, apport first needs to be enabled to collect core dumps: +#### Core dump via apport +I did not find an easy way to use apport. Anyway, the systemd approach works +fine. So remove apport, install systemd-coredump, and verify it is the new +coredump handler: ``` -sudo systemctl enable apport.service -``` - -and [needs to be enabled](https://wiki.ubuntu.com/Apport#How_to_enable_apport). -Then, show a list of core dumps using - -``` -sudo apport-cli +sudo systemctl stop apport +sudo systemctl mask --now apport +sudo apt install systemd-coredump +# Verify this changed the core pattern to a pipe to systemd-coredump +sysctl kernel.core_pattern ``` diff --git a/docker/debug_core_image.sh b/docker/debug_core_image.sh new file mode 100755 index 0000000000..08d1719964 --- /dev/null +++ b/docker/debug_core_image.sh @@ -0,0 +1,56 @@ +#!/bin/bash + +if [ $# -ne 3 ]; then + echo "usage: $0 " + exit 1 +fi + +die() { + echo $1 + exit 1 +} + +IMAGE=$1 +COREDUMP=$2 +SOURCES=$3 + +set -x + +# the image/build_oai builds in cmake_targets/ran_build/build, so source +# information is relative to this path. In case the user did not compile on +# their computer, this directory will not exist. still allow to find it by +# creating it +BUILD_DIR=$SOURCES/cmake_targets/ran_build/build +mkdir -p $BUILD_DIR || die "cannot create $BUILD_DIR: is $SOURCES valid?" + +# check if coredump is valid file +[ -f $COREDUMP ] || die "no such file: $COREDUMP" + +# check if image exists, and determine type (gnb, nr-ue) for correct invocation +# of gdb +docker image inspect $IMAGE > /dev/null || exit 1 +if [ $(grep "oai-gnb:" <<< $IMAGE) ] || [ $(grep "oai-gnb-aerial:" <<< $IMAGE) ]; then + EXEC=bin/nr-softmodem + TYPEPATH=oai-gnb +elif [ $(grep "oai-nr-ue:" <<< $IMAGE) ]; then + EXEC=bin/nr-uesoftmodem + TYPEPATH=oai-nr-ue +elif [ $(grep "oai-enb:" <<< $IMAGE) ]; then + EXEC=bin/lte-softmodem + TYPEPATH=oai-enb +elif [ $(grep "oai-lte-ue:" <<< $IMAGE) ]; then + EXEC=bin/lte-uesoftmodem + TYPEPATH=oai-lte-ue +else + die "cannot determine if image is gnb or nr-ue: must match \"oai-gnb:\" or \"oai-nr-ue:\"" +fi + +# run gdb inside a container. We mount the coredump and the sources inside the +# container, and run gdb with the core dump, the correct executable, and using +# the source directory to show correct line numbers +docker run --rm -it \ + -v $COREDUMP:/tmp/coredump \ + -v $SOURCES:/opt/$TYPEPATH/src \ + --entrypoint bash \ + $IMAGE \ + -c "gdb --dir=src/cmake_targets/ran_build/build $EXEC /tmp/coredump"