maki/openairinterface5g
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/data_recording_app_optimization_part1' into integration_2026_w17 (!4061)

Browse Source 
T-Tracer & Data Recording v1.1: UL PHY trace modularization and recording
enhancements

Based on the plan with @schmidtr  and @roux-cedric, we agreed to enhance
the data recording in two phases:

- Phase 1: Update Data Recording framework to get All internal Massive
  Updates and enhancements through the whole framework public
- Phase2: Restructure T-Tracer Apps and remove all Mems in gNB and UE
  Softmodems to Get data on T-tarcer Apps for all activated Messages
  symbol-by-symbol, and related Meta-data, then do the aggregation on
  the T-tracer App. So, there is No Memory Creation at all on the 5G NR
  Stack.

This Merge Request is for Phase 1. We address the comments given in: !3632

Summary: Refactor T-Tracer instrumentation for UL PHY data capture and
improve the data recording application (v1.0 → v1.1). This MR extracts
inline T() macro calls into modular wrapper functions, adds Unix timestamp
fields to trace messages, implements a 4-state shared memory protocol for
gNB/UE tracers, and delivers an improved synchronized recording architecture
in Python.

Changes:
- T-Tracer: Modular UL PHY trace wrappers
  - Add T_messages_creator.c/.h with 7 wrapper functions (log_ul_fd_dmrs,
    log_ul_fd_chan_est_dmrs_pos, log_ul_fd_pusch_iq,
    log_ul_fd_chan_est_dmrs_interpl, log_ul_payload_rx_bits,
    log_ul_payload_tx_bits, log_ul_scrambled_tx_bits) and a static
    log_ul_common() helper
  - Register T_messages_creator.c in the T library CMake build
  - Update T_messages.txt definitions: Remove string timestamps and get
    time stamp directly from sending_time of T-macro.
  - PHY: Replace inline T() calls with wrappers
  - gNB (nr_ulsch_demodulation.c, nr_ul_channel_estimation.c,
    phy_procedures_nr_gNB.c): replace inline T() calls with
    T_messages_creator wrappers
  - UE (nr_ulsch_coding.c, nr_ulsch_ue.c): replace inline T() calls with
    log_ul_payload_tx_bits and log_ul_scrambled_tx_bits wrappers
  - Remove deprecated log_tools.c/.h and its CMake reference

- T-Tracer gNB & UE apps: Enhancements
  - Implement 4-state shared memory protocol (WAIT → CONFIG → RECORD → STOP)
    for coordination with the data recording app
  - Add event_trace_msg_ul_data struct with 27 UL metadata fields
  - Refactor event loop to poll()-based with proper signal handling
  - Separate project IDs (gNB=2335, UE=2336) and ftok paths for independent
    shared memory segments
  - Update shared memory sizing: Limit number of records to
    NUM_MESSAGES_PER_SLOT=5, 100 slots

- Data Recording App v1.1
  - Improved architecture with thread pool and barrier synchronization
  - Frame/slot-based message grouping with per-slot validation
  - atexit/signal-based cleanup for graceful shared memory detach
  - Delete legacy data_recording_app_v1.0.py

- Library improvements
  - config_interface.py: remove legacy YAML config reader, streamline to
    JSON-only
  - shared_memory_interface.py: new module for shared memory
    attach/read/protocol
  - wireless_parameters_mapper.py: new module for NR parameter mapping
  - data_recording_messages_def.py: updated message definitions for
    27-field struct
  - sigmf_interface.py, sync_service.py, common_utils.py: remove unused
    imports and dead code

- Documentation
  - Update data_recording.md for v1.1 architecture and usage
This commit is contained in:
Jaroslava Fiedlerova
2026-04-24 14:09:41 +02:00
parent 7487638d72 bb11b8eafe
commit 6b06bda933
28 changed files with 2328 additions and 2442 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/data_recording.md
+64 -46
View File
@@ -50,10 +50,11 @@ python3 -m pip install -r common/utils/data_recording/requirements.txt
The Data Recording application provides configuration file in [JSON](http://www.json.org/) format. It is stored in [common/utils/data_recording/config/config_data_recording.json](../common/utils/data_recording/config/config_data_recording.json) folder. The main parameters are:
- **data_storage_path**: Path to directory for data storage
- **num_records**: Number of requested data records in slots
- **num_records**: Number of requested data records in slots. The maximum number of num_records depends on the memory size. If the user would like to record large number of records, user needs to increase the memory size defined here (common/utils/T/tracer/shared_memory_config.h)
- **t_tracer_message_definition_file**: T-Tracer message definition file
- **parameter_map_file**: Parameter mapping dictionary (OAI parameters to standardized metadata). It is located here: [common/utils/data_recording/config/wireless_link_parameter_map.yaml](../common/utils/data_recording/config/wireless_link_parameter_map.yaml)
- **start_frame_number**: It can be used to start the recording from a specific frame, but it is not yet supported.
- **common_meta_data**: Common metadata parameters that are not read from the system yet, but shall be included into the SigMF metadata such as the sampling rate, the bandwidth, and the clock reference of USRP.
- **base_station**:
- requested_tracer_messages: Requested base station data traces. The supported messages are:
@@ -62,16 +63,18 @@ The Data Recording application provides configuration file in [JSON](http://www.
- GNB_PHY_UL_FD_CHAN_EST_DMRS_POS
- GNB_PHY_UL_FD_CHAN_EST_DMRS_INTERPL
- GNB_PHY_UL_PAYLOAD_RX_BITS
- tracer_service_baseStation_address: The IP address and port number of base station server. If the Data Recording App and the Data Collection (T–Tracer) Service are running on the same server of base station, the IP address and port number are : `127.0.0.1:2021`
- meta_data: Additional base station metadata (additional fixed base station parameters that cannot be read from the system yet, but shall be included into the SigMF metadata)
- shared_mem_config: Shared Memory configuration such as in T-Tracer App of gNB
- **user_equipment**
- requested_tracer_messages: Requested user equipment data traces. The supported messages are:
- UE_PHY_UL_SCRAMBLED_TX_BITS
- UE_PHY_UL_PAYLOAD_TX_BITS
- tracer_service_userEquipment_address**: The IP address and port number of UE server.
- meta_data: Additional user equipment metadata (additional fixed UE parameters that are not read from the system yet, but shall be included into the SigMF metadata)
- **common_meta_data**: Common metadata parameters that are not read from the system yet, but shall be included into the SigMF metadata such as the sampling rate, the bandwidth, and the clock reference of USRP.
- **tracer_service_baseStation_address**: The IP address and port number of base station server. If the Data Recording App and the Data Collection (T–Tracer) Service are running on the same server of base station, the IP address and port number are : `127.0.0.1:2021`
- **tracer_service_userEquipment_address**: The IP address and port number of UE server.
- shared_mem_config: Shared Memory configuration such as in T-Tracer App of UE
**Notes**:
- The data can be recorded from base station and UE or from base station only or UE only. To disable the recording from any station, make the list of `requested_tracer_messages` empty such as: `requested_tracer_messages:[]`
@@ -79,54 +82,68 @@ The Data Recording application provides configuration file in [JSON](http://www.
The figure below illustrates an example of a JSON Data Recording App configuration file.
```json
{
"data_recording_config": {
"data_storage_path": "/home/user/workarea/oai_recorded_data/",
"data_file_format": "SigMF",
"enable_saving_tracer_messages_sigmf": true,
"num_records": 5,
"t_tracer_message_definition_file": "../T/T_messages.txt",
"parameter_map_file": "config/wireless_link_parameter_map.yaml",
"start_frame_number": 5,
"base_station": {
"data_storage_path": "/home/user/workarea/oai_recorded_data/",
"data_file_format": "SigMF",
"enable_saving_tracer_messages_sigmf": true,
"num_records": 5,
"t_tracer_message_definition_file": "../T/T_messages.txt",
"parameter_map_file": "config/wireless_link_parameter_map.yaml",
"start_frame_number": 5,
"common_meta_data": {
"sample_rate": 61440000.0,
"bandwidth": 40000000.0,
"clock_reference": "external"
},
"nodes": [
{
"id": "gnb_0",
"type": "gnb",
"requested_tracer_messages": [
"GNB_PHY_UL_FD_PUSCH_IQ",
"GNB_PHY_UL_FD_DMRS",
"GNB_PHY_UL_FD_PUSCH_IQ",
"GNB_PHY_UL_FD_DMRS",
"GNB_PHY_UL_FD_CHAN_EST_DMRS_POS",
"GNB_PHY_UL_FD_CHAN_EST_DMRS_INTERPL",
"GNB_PHY_UL_PAYLOAD_RX_BITS"
],
"meta_data":{
"GNB_PHY_UL_PAYLOAD_RX_BITS"],
"tracer_service_address": "127.0.0.1:2021",
"meta_data": {
"num_tx_antennas": 1,
"num_rx_antennas": 1,
"tx_gain": 48.0,
"rx_gain": 30.0,
"hw_type": "USRP X410",
"hw_subtype": "ZBX",
"seid": "328AB35"
},
"shared_mem_config": {
"read_path": "/tmp/gnb_app1",
"write_path": "/tmp/gnb_app2",
"project_id": 2335
}
},
"user_equipment": {
{
"id": "ue_0",
"type": "ue",
"requested_tracer_messages": [
"UE_PHY_UL_SCRAMBLED_TX_BITS",
"UE_PHY_UL_PAYLOAD_TX_BITS"
],
"meta_data":{
"UE_PHY_UL_SCRAMBLED_TX_BITS",
"UE_PHY_UL_PAYLOAD_TX_BITS"],
"tracer_service_address": "127.0.0.1:2023",
"meta_data": {
"num_tx_antennas": 1,
"num_rx_antennas": 1,
"tx_gain": 48.0,
"rx_gain": 40.0,
"rx_gain": 30.0,
"hw_type": "USRP X410",
"hw_subtype": "ZBX",
"seid": "323F75F"
},
"shared_mem_config": {
"read_path": "/tmp/ue_app1",
"write_path": "/tmp/ue_app2",
"project_id": 2336
}
},
"common_meta_data": {
"sample_rate": 61440000.0,
"bandwidth": 40000000.0,
"clock_reference": "external"
},
"tracer_service_baseStation_address": "127.0.0.1:2021",
"tracer_service_userEquipment_address": "127.0.0.1:2023"
}
}
]
}
```
@@ -153,7 +170,7 @@ The following figure shows an example of Wireless Link Parameter Map Dictionary.
### Global Metadata
There are some metadata parameters that the user may need to change only once. Those parameters have been hard coded in the Data Recording App header.
[common/utils/data_recording/data_recording_app_v1.0.py](../common/utils/data_recording/data_recording_app_v1.0.py). Those are:
[common/utils/data_recording/data_recording_app_v1.1.py](../common/utils/data_recording/data_recording_app_v1.1.py). Those are:
- The global metadata such as: author, description, sigmf collection file prefix, datetime_offset, enable saving config Data Recording App in json file with recorded data, and name of signal generator (i.e. 5gnr_oai). The name of signal generator is used for parameter mapping dictionary (OAI parameters to standardized metadata).
- The mapping between supported OAI messages and file_name_prefix, scope, and description.
@@ -237,11 +254,11 @@ For the test simplicity, user can run the system in PHY-test mode. We will show
Run NR gNB Softmodem with USRP:
```
sudo ./cmake_targets/ran_build/build/nr-softmodem -O ./targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb.band78.sa.fr1.106PRB.1x1.usrpx410_3300MHz.conf --gNBs.[0].min_rxtxtime 6 --usrp-tx-thread-config 1 --phy-test -d --T_stdout 2 --T_nowait
sudo ./cmake_targets/ran_build/build/nr-softmodem -O ./targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb0.prs.band78.fr1.106PRB.usrpx310 --gNBs.[0].min_rxtxtime 6 --usrp-tx-thread-config 1 --phy-test -d --T_stdout 2 --T_nowait
```
Run NR gNB Softmodem in RF Simulation:
```
sudo ./cmake_targets/ran_build/build/nr-softmodem -O ./targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb.band78.sa.fr1.106PRB.1x1.usrpx410_3300MHz.conf --gNBs.[0].min_rxtxtime 6 --rfsim --rfsimulator.[0].serveraddr server --phy-test -d --T_stdout 2 --T_nowait
sudo ./cmake_targets/ran_build/build/nr-softmodem -O ./targets/PROJECTS/GENERIC-NR-5GC/CONF/gnb0.prs.band78.fr1.106PRB.usrpx310.conf --gNBs.[0].min_rxtxtime 6 --rfsim --phy-test --rfsimulator.[0].serveraddr server --T_stdout 2 --T_nowait
```
> **Note:** User needs to change the name of gNB config file to your gNB config file.
@@ -265,9 +282,9 @@ sudo ./cmake_targets/ran_build/build/nr-uesoftmodem --usrp-args "type=x4xx,addr=
```
Run OAI Soft UE in RF Simulation, assume the IP address of gNB server is `192.168.100.2`:
Run OAI Soft UE in RF Simulation, assume both gNB and UE are running on the same server in Simulation mode`:
```
sudo ./cmake_targets/ran_build/build/nr-uesoftmodem --rfsim --rfsimulator.[0].serveraddr 192.168.100.2 --phy-test -O ./targets/PROJECTS/GENERIC-NR-5GC/CONF/ue.conf --reconfig-file /home/user/workarea/rrc_files/reconfig.raw --rbconfig-file /home/user/workarea/rrc_files/rbconfig.raw --T_stdout 2 --T_nowait --T_port 2023
sudo ./cmake_targets/ran_build/build/nr-uesoftmodem --rfsim --phy-test -O ./targets/PROJECTS/GENERIC-NR-5GC/CONF/ue.conf --reconfig-file /home/user/workarea/openairinterface5g/reconfig.raw --rbconfig-file /home/user/workarea/openairinterface5g/rbconfig.raw --T_stdout 2 --T_nowait --T_port 2023
```
### Step3: Run Data Recording Application Services
@@ -286,7 +303,7 @@ Run gNB Data Collection service on one terminal:
```
Run UE Data Collection service on another terminal:
```
./t_tracer_app_ue -d ../T_messages.txt
./t_tracer_app_ue -d ../T_messages.txt -p 2023
```
Second, run main Data Recording Application:
@@ -294,7 +311,7 @@ Second, run main Data Recording Application:
Run the following python script:
```
python3 data_recording_app_v1.0.py
python3 data_recording_app_v1.1.py
```
The recorded data set will be stored in the configured path, assume `/home/user/workarea/oai_recorded_data/`.
@@ -372,8 +389,8 @@ The following figure shows an example of SigMF metadata file, for example for th
{
"global": {
"core:author": "Abdo Gaber",
"core:collection": "data-collection-rec-0-2025_06_19T11_19_00_037",
"core:dataset": "raw-ce-fd-data-rec-0-2025_06_19T11_19_00_037.sigmf-data",
"core:collection": "data-collection-rec-1-2026_04_21T14_14_07_008277523",
"core:dataset": "raw-ce-fd-data-rec-1-2026_04_21T14_14_07_008279596.sigmf-data",
"core:datatype": "cf32_le",
"core:description": "Frequency-domain raw channel estimates (at DMRS positions)",
"core:hw": "USRP X410",
@@ -381,12 +398,12 @@ The following figure shows an example of SigMF metadata file, for example for th
"core:num_channels": 1,
"core:recorder": "NI Data Recording Application for OAI",
"core:sample_rate": 61440000.0,
"core:sha512": "55ae5bc81b4812e68f6e1c195600f240fc3ca71113c732271ee42a5524da98636c3c7ac587c46fd87c6d1b9253ca6504a4b94eafba8f4065099096fc7721228c",
"core:sha512": "fce1c6e85e986939e8cf5a3937d0d2f08d9f27ab52ea4065d6e347beb5acf7017a4f9d200c70345f1724ff52d886403e2fddedce62c5c1de4ef895a941547e3d",
"core:version": "1.2.1"
},
"captures": [
{
"core:datetime": "2025_06_19T11:19:00.037",
"core:datetime": "2026_04_21T14:14:07.008279596",
"core:frequency": 3319680000.0,
"core:sample_start": 0,
"serialization:counts": [
@@ -431,10 +448,11 @@ The following figure shows an example of SigMF metadata file, for example for th
"5gnr": {
"cell_id": 0,
"cp_mode": "normal",
"frame": 785,
"frame": 916,
"frame_structure": "tdd",
"frequency_range": "fr1",
"link_direction": "uplink",
"num_rx_antennas": 1,
"num_slots": 1,
"num_tx_antennas": 1,
"pusch:content": "compliant",
@@ -445,7 +463,7 @@ The following figure shows an example of SigMF metadata file, for example for th
"pusch:num_layer": 1,
"pusch:num_ofdm_symbols": 13,
"pusch:num_prb": 50,
"pusch:payload_bit_pattern": "zeros",
"pusch:payload_bit_pattern": "random",
"pusch:start_ofdm_symbol": 0,
"pusch:start_prb": 0,
"pusch:transform_precoding": "disabled",