Speed up docker deployment in CI
- Using docker compose up --wait instead of a custom healthcheck script:
from: https://docs.docker.com/reference/cli/docker/compose/up/
--wait Wait for services to be running|healthy. Implies detached mode.
--wait-timeout Maximum duration in seconds to wait for the project to be running|healthy
- Speeding up the healthchecks by reducing the interval from 1-10s to
0.5s during start period
start_period creates a special period after the start of the container
where the healthcheck is executed every start_interval . During this
time the failures do not count towards the retries counter, but
successes change the status to healthy. This allows container to reach
healthy status much sooner.
- Using init: true to speed up docker compose down/stop
`init: true` initializes the docker container with tini or
alternative: tini is used to collect all the processes and correctly
pass the termination signal to them. This allows the docker container
to be stopped without waiting 10 seconds to kill the container
- Smallest example of this is:
services:
test-init:
image: ubuntu:jammy
command: sleep 10d
We use the same resource for CSI meas report and RSRP (judging by the
name of the functions), but the memory allocations need to be separate
to avoid a double free:
==2063066==ERROR: AddressSanitizer: heap-use-after-free on address 0x7c27d43e3ce8 at pc 0x000000fcd279 bp 0x7be7cc9e7120 sp 0x7be7cc9e7118
READ of size 8 at 0x7c27d43e3ce8 thread T6
#0 0x000000fcd278 in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:97
#1 0x000000fcd278 in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#2 0x000000fd6249 in SET_OF_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SET_OF.c:236
#3 0x000000fd6249 in SET_OF_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SET_OF.c:220
#4 0x000000fcd103 in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:91
#5 0x000000fcd103 in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#6 0x000000fc90c9 in CHOICE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE.c:185
#7 0x000000fc90c9 in CHOICE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE.c:158
#8 0x000000fcd103 in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:91
#9 0x000000fcd103 in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#10 0x000000fd6249 in SET_OF_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SET_OF.c:236
#11 0x000000fd6249 in SET_OF_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SET_OF.c:220
#12 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:88
#13 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#14 0x000000fc90c9 in CHOICE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE.c:185
#15 0x000000fc90c9 in CHOICE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE.c:158
#16 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:88
#17 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#18 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:88
#19 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#20 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:88
#21 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#22 0x0000009d78ae in ue_context_setup_request /home/richie/w/f1-refactor-ue-context/openair2/LAYER2/NR_MAC_gNB/mac_rrc_dl_handler.c:692
#23 0x00000083fee2 in rrc_add_nsa_user /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB_nsa.c:330
#24 0x000000886e8b in rrc_gNB_process_f1_setup_req /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB_du.c:444
#25 0x000000833942 in rrc_gnb_task /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:2766
#26 0x7fe7d5e28ee5 in asan_thread_start(void*) (/lib64/libasan.so.8+0x28ee5) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
#27 0x7fe7d527f1d3 in start_thread (/lib64/libc.so.6+0x711d3) (BuildId: 2b3c02fe7e4d3811767175b6f323692a10a4e116)
#28 0x7fe7d5301ceb in __GI___clone3 (/lib64/libc.so.6+0xf3ceb) (BuildId: 2b3c02fe7e4d3811767175b6f323692a10a4e116)
0x7c27d43e3ce8 is located 24 bytes inside of 40-byte region [0x7c27d43e3cd0,0x7c27d43e3cf8)
freed by thread T6 here:
#0 0x7fe7d5ee5bcb in free.part.0 (/lib64/libasan.so.8+0xe5bcb) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
#1 0x000000fd6249 in SET_OF_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SET_OF.c:236
#2 0x000000fd6249 in SET_OF_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SET_OF.c:220
#3 0x000000fcd103 in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:91
#4 0x000000fcd103 in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#5 0x000000fc90c9 in CHOICE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE.c:185
#6 0x000000fc90c9 in CHOICE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE.c:158
#7 0x000000fcd103 in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:91
#8 0x000000fcd103 in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#9 0x000000fd6249 in SET_OF_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SET_OF.c:236
#10 0x000000fd6249 in SET_OF_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SET_OF.c:220
#11 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:88
#12 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#13 0x000000fc90c9 in CHOICE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE.c:185
#14 0x000000fc90c9 in CHOICE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE.c:158
#15 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:88
#16 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#17 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:88
#18 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#19 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:88
#20 0x000000fcd01a in SEQUENCE_free /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE.c:69
#21 0x0000009d78ae in ue_context_setup_request /home/richie/w/f1-refactor-ue-context/openair2/LAYER2/NR_MAC_gNB/mac_rrc_dl_handler.c:692
#22 0x00000083fee2 in rrc_add_nsa_user /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB_nsa.c:330
#23 0x000000886e8b in rrc_gNB_process_f1_setup_req /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB_du.c:444
#24 0x000000833942 in rrc_gnb_task /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:2766
#25 0x7fe7d5e28ee5 in asan_thread_start(void*) (/lib64/libasan.so.8+0x28ee5) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
previously allocated by thread T6 here:
#0 0x7fe7d5ee6883 in calloc (/lib64/libasan.so.8+0xe6883) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
#1 0x000000a1ac53 in get_default_secondaryCellGroup /home/richie/w/f1-refactor-ue-context/openair2/LAYER2/NR_MAC_gNB/nr_radio_config.c:3969
#2 0x0000009d8913 in create_new_UE /home/richie/w/f1-refactor-ue-context/openair2/LAYER2/NR_MAC_gNB/mac_rrc_dl_handler.c:571
#3 0x0000009d8913 in ue_context_setup_request /home/richie/w/f1-refactor-ue-context/openair2/LAYER2/NR_MAC_gNB/mac_rrc_dl_handler.c:638
#4 0x00000083fee2 in rrc_add_nsa_user /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB_nsa.c:330
#5 0x000000886e8b in rrc_gNB_process_f1_setup_req /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB_du.c:444
#6 0x000000833942 in rrc_gnb_task /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:2766
#7 0x7fe7d5e28ee5 in asan_thread_start(void*) (/lib64/libasan.so.8+0x28ee5) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
Thread T6 created by T0 here:
#0 0x7fe7d5ede492 in pthread_create (/lib64/libasan.so.8+0xde492) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
#1 0x0000012a4ba3 in threadCreate /home/richie/w/f1-refactor-ue-context/common/utils/system.c:283
#2 0x0000006b5fef in itti_create_task /home/richie/w/f1-refactor-ue-context/common/utils/ocp_itti/intertask_interface.cpp:317
#3 0x000000435c5f in create_gNB_tasks /home/richie/w/f1-refactor-ue-context/executables/nr-softmodem.c:310
#4 0x000000435c5f in main /home/richie/w/f1-refactor-ue-context/executables/nr-softmodem.c:620
#5 0x7fe7d52115f4 in __libc_start_call_main (/lib64/libc.so.6+0x35f4) (BuildId: 2b3c02fe7e4d3811767175b6f323692a10a4e116)
#6 0x7fe7d52116a7 in __libc_start_main@@GLIBC_2.34 (/lib64/libc.so.6+0x36a7) (BuildId: 2b3c02fe7e4d3811767175b6f323692a10a4e116)
#7 0x0000004672d4 in _start (/home/richie/w/f1-refactor-ue-context/build/nr-softmodem+0x4672d4) (BuildId: 81450e9a0b832ca9e36b1984ecc232efde59d424)
Closes: #958
Direct leak of 512 byte(s) in 1 object(s) allocated from:
#0 0x7f37634e5e2b in realloc.part.0 (/lib64/libasan.so.8+0xe5e2b) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
#1 0x000000fbb9ad in encode_dyn_cb /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/per_encoder.c:23
#2 0x000000fc279e in asn_put_few_bits /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/asn_bit_data.c:220
#3 0x000000fc279e in asn_put_few_bits /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/asn_bit_data.c:192
#4 0x000000fc2ce5 in asn_put_many_bits /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/asn_bit_data.c:293
#5 0x000000fb8c33 in OCTET_STRING_encode_uper /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/OCTET_STRING_uper.c:306
#6 0x000000fd191a in SEQUENCE_encode_uper /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE_uper.c:403
#7 0x000000fd191a in SEQUENCE_encode_uper /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE_uper.c:403
#8 0x000000fcb140 in CHOICE_encode_uper /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE_uper.c:177
#9 0x000000fd191a in SEQUENCE_encode_uper /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE_uper.c:403
#10 0x000000fcb140 in CHOICE_encode_uper /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE_uper.c:177
#11 0x000000fcb140 in CHOICE_encode_uper /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_CHOICE_uper.c:177
#12 0x000000fd191a in SEQUENCE_encode_uper /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/constr_SEQUENCE_uper.c:403
#13 0x000000fbc1ed in uper_encode /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/uper_encoder.c:27
#14 0x000000fbc73a in uper_encode_to_new_buffer /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/uper_encoder.c:85
#15 0x000000e10fbd in do_RRCReconfiguration /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/MESSAGES/asn1_msg.c:586
#16 0x000000805712 in rrc_gNB_encode_RRCReconfiguration /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:652
#17 0x000000893860 in nr_rrc_f1_ho_acknowledge /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB_mobility.c:280
#18 0x0000008079ac in rrc_CU_process_ue_context_setup_response /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:2139
#19 0x0000008330d8 in rrc_gnb_task /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:2771
#20 0x7f3763428ee5 in asan_thread_start(void*) (/lib64/libasan.so.8+0x28ee5) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
During refactoring, I had asan errors in the next line. It took me some
time to figure out that a DRB ID 0 is not valid (cf.
get_next_available_drb_id() and get_drb(), they enforce this).
Direct leak of 48 byte(s) in 1 object(s) allocated from:
#0 0x7fea71ee6f0b in malloc (/lib64/libasan.so.8+0xe6f0b) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
#1 0x0000007f29c0 in malloc_or_fail /home/richie/w/f1-refactor-ue-context/common/utils/utils.h:86
#2 0x0000007f29c0 in cuup_notify_reestablishment /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:872
#3 0x000000808002 in rrc_CU_process_ue_context_setup_response /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:2126
#4 0x0000008330d8 in rrc_gnb_task /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:2770
#5 0x7fea71e28ee5 in asan_thread_start(void*) (/lib64/libasan.so.8+0x28ee5) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
Direct leak of 32 byte(s) in 1 object(s) allocated from:
#0 0x7f672d6e5e2b in realloc.part.0 (/lib64/libasan.so.8+0xe5e2b) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
#1 0x000000fbb4ed in encode_dyn_cb /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/per_encoder.c:23
#2 0x000000fbbeeb in _uper_encode_flush_outp /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/uper_encoder.c:126
#3 0x000000fbbeeb in uper_encode /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/uper_encoder.c:36
#4 0x000000fbc27a in uper_encode_to_new_buffer /home/richie/w/f1-refactor-ue-context/build/openair2/RRC/LTE/MESSAGES/uper_encoder.c:85
#5 0x00000081037c in handle_ueCapabilityInformation /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:1549
#6 0x000000813ced in rrc_gNB_decode_dcch /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:1902
#7 0x0000008334f2 in rrc_gnb_task /home/richie/w/f1-refactor-ue-context/openair2/RRC/NR/rrc_gNB.c:2741
#8 0x7f672d628ee5 in asan_thread_start(void*) (/lib64/libasan.so.8+0x28ee5) (BuildId: 7f1aa7e2e600e8c9d54ce6e3d36f3d31bfe7949a)
Move the filling of DRBs to set up for UE context setup/modification
request into the corresponding functions, and use malloc() deliberately
to avoid memory leaks. In fact fill_drb_to_be_setup_from_e1_resp()
allocates memory dynamically (and has to do that), so the easiest is to
do the same for DRBs. The most straight-forward way seems to be to then
dynamically allocate all memory for F1 UE context setup/modification
request messages, and free with the corresponding free functions,
which removes a couple of memory leaks.
Similarly to the commit "Use F1 UE context setup/modif Request in
stack", we do both UE context setup/modification response at once
because of some functions that are shared by the MAC message handler.
Use the new, tested F1 UE context setup request and modification
requests in the stack.
We have to do both at once, because there are some functions that are
used by handlers for both messages, notably
- handle_ue_context_drbs_setup()
- handle_ue_context_srbs_setup()
(+ and some functions called by those two). For instance, the first uses
the (new/old) type f1ap_drb_to_be_setup_t/f1ap_drb_to_setup_t which is
shared. The alternative would be to duplicate the functionality
temporarily, and I want to avoid unnecessary code churn.
Reintroduce separate message types to simplify the encoding/decoding,
including shorter names. The old types will be removed once everything
has been migrated to the new encoder/decoder, which should ensure that
everything is properly migrated to the unit-tested encoder/decoders.
Also, for each type of F1 message (F1 UE context setup request,
response, modification request, ...) introduce a separate type to avoid
to mistakenly use wrong fields, which can lead to bugs. In this regard,
the DL PDCP SN bit size in the DRB-ToBeSetup list of the UE context
setup request is optional, as it will also be used in the UE context
modification request DRB SetupMod list (which avoids quite some code
duplication). The encoder checks that this IE is present, though.
The previous condition was wrong: if a UE did not have the assoc_id of
the DU that connected (i.e., it was connected to another DU then the one
disconnecting), it was marked for deletion. Instead, skip those UEs.
This modification aims to improve health status detection by using PRACH IO as
an indicator of the gNB's status (container is marked as "healthy" when the
PRACH IO value in gNB nrL1_stats.log exceeds 0.0 dB.). The new condition accounts
for scenarios where the container may not exit or produce error codes (e.g.,
“sleep” logs or "error code overflow" messages).
Robert Schmidt