diff --git a/common/utils/T/DOC/T.md b/common/utils/T/DOC/T.md new file mode 100644 index 0000000000..acc06a057f --- /dev/null +++ b/common/utils/T/DOC/T.md @@ -0,0 +1,19 @@ +# T tracer + +The T tracer is sort of a *framework* to debug and monitor the softmodem. + +It is made of two main parts: + +* an events' collector integrated to the real-time processing, +* a separate set of programs to receive, record, display, replay and + analyse the events sent by the collector. + +## Documentation + +* [Basic usage](./T/basic.md) +* [Record](./T/record.md) +* [Replay](./T/replay.md) +* [Multiple tracers](./T/multi.md) +* [MAC PDUs and wireshark](./T/wireshark.md) +* [The *enb* tracer](./T/enb.md) +* [The *to_vcd* tracer](./T/to_vcd.md) diff --git a/common/utils/T/DOC/T/basic.md b/common/utils/T/DOC/T/basic.md new file mode 100644 index 0000000000..6bac0c5801 --- /dev/null +++ b/common/utils/T/DOC/T/basic.md @@ -0,0 +1,90 @@ +# Basic usage of the T + +## Compilation + +### softmodem side + +Simply call `build_oai` the usual way, for example `./build_oai --eNB -w USRP`. +The T tracer is compiled in by default. + +### tracer side + +Go to the directory `common/utils/T/tracer` and do `make`. This will locally +compile all tracer executables, and place them in `common/utils/T/tracer`. If +you wish to keep source and build separate, go to your existing build directory +(typically `cmake_targets/ran_build/build`), and do `make T_tools` (or `ninja +T_tools`). After that, the executables will be in +`/common/utils/T/tracer`. + +In case of failure with one of the following errors: + +```shell +/usr/bin/ld: cannot find -lXft +``` +or +```shell +-- Checking for module 'xft' +-- No package 'xft' found +CMake Error at /usr/share/cmake-3.16/Modules/FindPkgConfig.cmake:463 (message): + A required package was not found +``` +Run: + +```shell +sudo apt-get install libxft-dev +``` + +## Run the softmodem + +Run the softmodem with the option `--T_stdout 2` and it will wait for a +tracer to connect to it before processing. (This option is confusing +and may change in the future.) + +```shell +cd cmake_targets/ran_build/build +sudo ./lte-softmodem -O [configuration file] --T_stdout 2 +``` + +Additional options can be passed to the softmodem. + +The option `--T_nowait` lets the processing start immediately without +first waiting for a tracer. + +The option `--T_port [port]` changes the default port used by the +softmodem to wait for a tracer. The default port is 2021. + +The option `--T_dont_fork` allows one to use gdb to debug problems +with the softmodem. Note that you then may have some "zombie" +processes after crashes, in which case you can run +`sudo killall -9 lte-softmodem` to get rid of them (`lte-softmodem` +to be replaced by the program you trace, like `lte-softmodem-nos1` +or `oaisim`). + +The option `--T_stdout` also accepts values 0 (to disable output +on the terminal and only use the T tracer) and 1 (to disable +the T tracer and only output on the terminal). The default is +1. + +## Run a tracer + +Go into the directory `common/utils/T/tracer` (or +`cmake_targets/ran_build/build/common/utils/T/tracer`) and run for example the +`enb` tracer: + +```shell +./enb -d ../T_messages.txt +``` + +To trace a remote program, use the `-ip` option. For example, +if you want to trace a program running on `192.168.12.148` do: + +```shell +./enb -d ../T_messages.txt -ip 192.168.12.148 +``` + +A graphical user interface will appear with which you can interact to +control the monitoring. See [there](./enb.md) for more documentation +about the `enb` tracer. + +It is possible to run several tracers at the same time. See +[here](./multi.md). diff --git a/common/utils/T/DOC/T/enb.md b/common/utils/T/DOC/T/enb.md new file mode 100644 index 0000000000..5d78503bfd --- /dev/null +++ b/common/utils/T/DOC/T/enb.md @@ -0,0 +1,3 @@ + +![enb_trace](./enb_trace.png) +[source file of this image](./enb_trace.odp) diff --git a/common/utils/T/DOC/T/enb_trace.odp b/common/utils/T/DOC/T/enb_trace.odp new file mode 100644 index 0000000000..0bfa2e8069 Binary files /dev/null and b/common/utils/T/DOC/T/enb_trace.odp differ diff --git a/common/utils/T/DOC/T/enb_trace.png b/common/utils/T/DOC/T/enb_trace.png new file mode 100644 index 0000000000..338f3e7210 Binary files /dev/null and b/common/utils/T/DOC/T/enb_trace.png differ diff --git a/common/utils/T/DOC/T/example.raw b/common/utils/T/DOC/T/example.raw new file mode 100644 index 0000000000..2273e3d0d6 Binary files /dev/null and b/common/utils/T/DOC/T/example.raw differ diff --git a/common/utils/T/DOC/T/multi.md b/common/utils/T/DOC/T/multi.md new file mode 100644 index 0000000000..db34103c38 --- /dev/null +++ b/common/utils/T/DOC/T/multi.md @@ -0,0 +1,57 @@ +# Multiple tracers + +To connect several tracers to the same process, first launch the `multi` +tracer. + +Run as: + +```shell +cd common/utils/T/tracer +./multi -d ../T_messages.txt +``` + +Then connect other tracers (`enb`, `textlog`, `to_vcd`, etc.) to the +`multi` tracer. + +For example (if you debug the `lte-softmodem`) in one terminal, run: + +```shell +cd common/utils/T/tracer +./enb -d ../T_messages.txt -p 2022 +``` + +That will launch the GUI tool to analyze the proper running of the +softmodem. + +And in another terminal, run: + +```shell +cd common/utils/T/tracer +./to_vcd -d ../T_messages.txt -p 2022 -vcd -o /tmp/openair_dump_eNB.vcd +``` + +That will dump the VCD traces generated by the softmodem to the +file `/tmp/openair_dump_eNB.vcd`. + +The `multi` tracer waits for other tracers to connect to the port 2022. +So you pass `-p 2022` to a tracer so that it connects to the `multi` +tracer. This port can be changed with the `-lp` option. + +For example to listen to port 4321, run `multi` as: + +```shell +./multi -d ../T_messages.txt -lp 4321 +``` + +Then you would pass `-p 4321` to any tracer for it to connect to `multi`. + +As a reminder, to trace `lte-softmodem`, you run it as: + +```shell +sudo ./lte-softmodem -O config_file.conf --T_stdout 0 +``` + +The important part is `--T_stdout 0`. + +The order to launch the various programs (`multi`, the softmodem, other +tracers) is not important. diff --git a/common/utils/T/DOC/T/record.md b/common/utils/T/DOC/T/record.md new file mode 100644 index 0000000000..c2e58e1e20 --- /dev/null +++ b/common/utils/T/DOC/T/record.md @@ -0,0 +1,34 @@ +# Record + +First, read the [basic usage](./basic.md) to compile things. + +To record, you use the `record` tracer. + +To send a trace to Eurecom, you run (unless we specifically ask you +to activate/deactivate specific traces): + +```shell +./record -d ../T_messages.txt -o record.raw -ON -off VCD -off HEAVY -off LEGACY_GROUP_TRACE -off LEGACY_GROUP_DEBUG +``` + +And then you run the program to trace (`lte-softmodem`, `oaisim`, whatever) +as explained in [basic usage](./basic.md). + +To stop the recording, you simply press `control+c` to end `record`. + +You send to Eurecom the file `record.raw`. The file `T_messages.txt` is not needed. + +To get a list of options, run: + +```shell +./record -h +``` + +The options `-ON`, `-OFF`, `-on` and `-off` are processed in order. +In the example above we first activate all the traces (`-ON`) and then +deactivate specific ones (more precisely, in this particular case: +the *groups* of traces `VCD`, `HEAVY`, `LEGACY_GROUP_TRACE` and +`LEGACY_GROUP_DEBUG`). + +You can then [replay](./replay.md) the recorded file or send it to others so they +can analyse things. diff --git a/common/utils/T/DOC/T/replay.md b/common/utils/T/DOC/T/replay.md new file mode 100644 index 0000000000..2390d177f6 --- /dev/null +++ b/common/utils/T/DOC/T/replay.md @@ -0,0 +1,39 @@ +# Replay + +First, read the [basic usage](./basic.md) to compile things. + +Then, read [record](./record.md) to know how to record a trace. + +To replay, you use the `replay` program. It will act +as a *tracee* (`lte-softmodem`, `oaisim`, etc.). + +Then you use your usual tracer, eg. `enb` or `textlog`. + +Since the T is constantly evolving you may need to extract +`T_messages.txt` contained in a trace to be able to process +the trace. + +## Example + +Download [example.raw](./example.raw). This example contains: +* MAC PDUs as sent and received by the eNodeB +* RRC info messages the eNodeB produced. + +First step is to extract `T_messages.txt` from this trace. + +```shell +./extract_config -i example.raw > extracted_T_messages.txt +``` + +Then you use `replay` to act as a regular *tracee*: + +```shell +./replay -i example.raw +``` + +And then, while `replay` is still running, you use the tracer +you want, for example the `textlog` tracer: + +```shell +./textlog -d extracted_T_messages.txt -no-gui -ON -full +``` diff --git a/common/utils/T/DOC/T/to_vcd.md b/common/utils/T/DOC/T/to_vcd.md new file mode 100644 index 0000000000..faec8bd47f --- /dev/null +++ b/common/utils/T/DOC/T/to_vcd.md @@ -0,0 +1,21 @@ +# to_vcd + +This tracer is used to dump a VCD trace of the softmodem. + +The basic usage is: + +```shell +cd common/utils/T/tracer +./to_vcd -d ../T_messages.txt -vcd -o /tmp/openair_dump_eNB.vcd +``` + +Apart from VCD specific traces, you can dump to VCD other T traces. +Use `-b` and `-l` options. For example, to only log DLSCH scheduler VCD traces, use: +```shell +./to_vcd -d ../T_messages.txt -o /tmp/openair_dump_eNB.vcd -b VCD_FUNCTION_SCHEDULE_DLSCH value mac_schedule_dlsch +``` + +For more help, run: +```shell +./to_vcd -h +``` diff --git a/common/utils/T/DOC/T/wireshark.md b/common/utils/T/DOC/T/wireshark.md new file mode 100644 index 0000000000..7ef7780bdd --- /dev/null +++ b/common/utils/T/DOC/T/wireshark.md @@ -0,0 +1,158 @@ +# MAC PDUs and wireshark + +First, read the [basic usage](./basic.md) to compile things. + +It is possible to use `wireshark` to analyse MAC PDUs for UEs, +MIBs, SIBs, and random accesses seen by the eNodeB. + +## Live usage + +Launch wireshark and listen on the local interface (lo). +Set the filter to `udp.port==9999` and read below +for configuration. + +Run the softmodem with the correct command line arguments +(see [basic usage](./basic.md)). + +Then run: + +```shell +./macpdu2wireshark -d ../T_messages.txt -live +``` + +For other options of `macpdu2wireshark`, run: + +```shell +./macpdu2wireshark -h +``` + +You can for example disable MIB and SIB logging, or log only +a given number of them. + +## Record and replay + +You need to record some specific traces. As of 2018-10-30 +you have to use the group: +* `WIRESHARK` + +That is, run: + +```shell +./record -d ../T_messages.txt -o /tmp/record.raw -on WIRESHARK +``` + +With a recorded trace you then use `macpdu2wireshark`. + +## Example + +Download [example.raw](example.raw). This example contains +MIBs, SIBs and a connection made by an UE followed by its +disconnection. + +First step is to extract `T_messages.txt` from this trace. + +```shell +./extract_config -i example.raw > extracted_T_messages.txt +``` + +Then you run `wireshark`, capture on `lo` and set the +filter to `udp.dstport==9999`. + +While `wireshark` is running, you do: + +```shell +./macpdu2wireshark -d extracted_T_messages.txt -i example.raw -no-mib -no-sib +``` + +In `wireshark` you should see something similar to the following: + +![wireshark window with dissected MAC PDUs](wireshark.png) + +## Configure wireshark for LTE + +To dissect the packets, you need to configure `wireshark`. + +In the menu, choose `Edit->Preferences`. + +In the preference window, unroll `Protocols`. + +Go to `UDP` and activate `Try heuristic sub-dissectors first`. + +Depending on your version of `wireshark`, you may need to go to `MAC-LTE` +and activate `Try Heuristic LTE-MAC over UDP framing` if this option +is present. + +For newer versions (after 2.0 as says wireshark's documentation), +heuristic activation is moved to the `Enabled Protocols` window +(in the menu `Analyze`). In the `enabled protocols` window, search +for `lte` and select `mac_lte_udp` to have `MAC-LTE over UDP`. + +You may also want to configure other parameters in `MAC-LTE`, +`RLC-LTE`, `PDCP-LTE`, and maybe others. Check the documentation +of `wireshark`. + +It is advised to use a recent version of wireshark (say more than 2.4). + +## Configure wireshark for NR + +Use a recent version of wireshark. The steps below were done using +version 3.3.2. Maybe some options are different for your version +of wireshark. Adapt as necessary. + +First, do the previous steps for LTE. + +Then, in the menu, choose `Edit->Preferences`. + +In the preference window, unroll `Protocols`. + +Go to `MAC-NR`. Select both options (`Attempt to decode BCCH, PCCH and CCCH data using NR RRC dissector` +and `Attempt to dissect LCID 1-3 as srb1-3`). + +For `Source of LCID -> drb channel settings` choose option `From static table`. +Then click the `Edit...` button of `LCID -> DRB Mappings Table`. + +In the new window, click on `+`. Choose LCID `4`, DRBID `1`, UL RLC Bearer +Type `AM, SN Len=18`, same thing for DL RLC Bearer Type. + +Then click OK. + +Now, go to `RLC-NR`. Select `Call PDCP dissector for SRB PDUs`. + +For `Call PDCP dissector for UL DRB PDUs` choose `18-bit SN`. Same +for DL. + +Select `Call RRC dissector for CCCH PDUs`. + +You don't need to select `May see RLC headers only` and +`Try to reassemble UM frames`. + +Now, go to `PDCP-NR`. Select what you want in there. It's good +to select `Show uncompressed User-Plane data as IP`. Also good +to select `Show unciphered Signalling-Plane data as RRC`. +For `Do sequence number analysis` it can be good to use +`Only-RLC-frames` but anything will do. We don't use ROHC so you +don't need to select `Attempt to decode ROHC data`. And the +layer info to show depends on what you want to analyse. +`Traffic Info` is a good choice. + +You are done with the preferences. You can click OK. + +Then, in the menu `Analyze`, choose `Enabled Protocols...`. In the new window +search for `nr` and select `mac_nr_udp` to have `MAC-NR over UDP`. + +And that's it. Maybe other settings can be changed, but those steps should be +enough for a start. + +Note: not all versions of Wireshark correctly dissect encapsulated packets. For instance, we observed that Wireshark v3.4.8 did not call the PDCP dissector for a PDCP PDU split in two RLC SDUs. On the other hand, Wireshark v3.7.0 showed it just fine. + +## Alternative to wireshark + +While using `macpdu2wireshark` with the `-live` option, it is possible to use `tcpdump` instead of `wireshak` to record MAC PDUs and also other types of traffic (sctp, gtp) in the same output pcap file. + +A "catch all" solution is to run: + +``` +sudo tcpdump -n -iany "udp port 9999 or udp port 2152 or sctp or port 38462 or port 38472 or port 38412 or port 2153" -w /tmp/rec.pcap +``` + +(This is an example, other solutions are also possible.) diff --git a/common/utils/T/DOC/T/wireshark.png b/common/utils/T/DOC/T/wireshark.png new file mode 100644 index 0000000000..15b138be7d Binary files /dev/null and b/common/utils/T/DOC/T/wireshark.png differ diff --git a/doc/README.md b/doc/README.md index cbe97a168b..776b217248 100644 --- a/doc/README.md +++ b/doc/README.md @@ -75,7 +75,7 @@ Legacy unmaintained files: ## General -- The [T tracer](https://gitlab.eurecom.fr/oai/openairinterface5g/-/wikis/T/): a generic tracing tool (VCD, Wireshark, GUI, to save for later, ...) +- The [T tracer](../common/utils/T/DOC/T.md): a generic tracing tool (VCD, Wireshark, GUI, to save for later, ...) - [OPT](../openair2/UTIL/OPT/README.txt): how to trace to wireshark - The [configuration module](../common/config/DOC/config.md) - The [logging module](../common/utils/LOG/DOC/log.md)