The bug was:
connect a phone, disconnect, repeat five times. The fifth time the phone
does not reconnect because there is no free RA process.
When an UE is removed, we have to free associated RA processes.
Before this commit, when the gNB crashes, the eNB keeps running.
Same, if the eNB crashes, the gNB keeps running.
In a far past when this happened, the other program (the one not
crashing) was forced to exit.
Then this behavior was changed, for some reason.
But the code was not finished, so now we have a system in an inconsistent
state.
So either we accept that the connection between eNB and gNB can break,
that is one of the programs crashes, and we clean the state of the
program that keeps running. But this is a complex work and it will
surely not survive very long, because someone will change something
in the code later that will break this complex behavior.
Or, simpler, we go back to initial behavior, which is: the program that
did not crash does actually exit when the other crashes.
This commit provides the second solution.
It can easily be reverted whenever someone wants to implement the complex
solution.
We used to set 'does_nr' of an UE if we detect ENDC supported in the
UE capabilities.
Then we activate NR measurements if 'does_nr' is true.
The problem is that if the eNB is not connected to a gNB but the UE
reports some NR measurements (because a gNB is running somewhere near)
then the eNB will crash when starting the switch to NR.
A (quick) solution is to set 'does_nr' only if there is a gNB connected to
the eNB.
Maybe not the best solution. To be changed if needed.
This code has not been tested with a COTS UE. We need SRB1/2 (so we need SA)
to test.
The activation of security and integrity for SRB1 will need more code
(new functions) because it happens after some time, not at creation time
of the PDCP entity and there is no way to do this with current code.
We do some sanity checks for incoming rlc control packets. One of them
is to check that so_start is not > so_end when present.
But when 'range' is present and bigger than 1 it means that so_start refers
to one PDU and so_end to another one. So we may well have so_start > so_end.
This commit fixes that and reorganizes a bit the code to do the check before
processing and rejecting the PDU if the values are not correct. (Before this
commit we were NACKing the whole PDUs if so_start > so_end.)
The discard timer is used by the PDCP layer to discard packets that were not
transmitted to RLC after some time.
We don't really need it for the moment, maybe never for the default
bearer which is a best-effort bearer passing all kind of IP traffic.
Maybe for voice over IP kind of traffic it's important, or realtime
applications.
I ran an iperf UDP uplink test and saw no error on the PHY layer but still
lots of packets lost in the iperf logs, which was very surprising. By
analyzing the logs in wireshark I saw that, with the current scheduler,
an IP packet sent by the UE took 40ms to be sent piece by piece to the
gNB, and then the next uplink IP packet was never sent at all because
the discard timer was expired, leading to the high count of packets lost
in the iperf logs.
So let's put infinity for the moment for the discard timer. From my
understanding of the moment it's not a problem. If for some reason we
should set it to some non-infinity value, then we can change later.
Implement data transfer (38.331 5.2):
- transmit operation (in nr_pdcp_entity_recv_sdu)
- receive operation (in nr_pdcp_entity_recv_pdu)
- t-Reordering expiration
To keep track of time the function nr_pdcp_wakeup_timer_thread has been
added and is called by the gNB's MAC scheduler via nr_pdcp_tick.
(The UE has not been modified.)
A thread has been introduced to set time of all the existing PDCP
entities in the system. This probably does not scale well with many
PDCP entities (many UEs connected to the gNB). To be changed at some
point when needed. Hopefully, with a few UEs this mechanism won't be
too problematic (ie. won't cause realtime problems).
The problem with time in PDCP is that SDUs are kept if some are
missing, to do reordering. A timer is started and when it expires
we have to transmit the SDUs even if some are still missing. We
want to do that as soon as possible. So we have to check timers
often (every millisecond).
In RLC the time is set when there is some activity (SDU or PDU
received). In PDCP we cannot wait for this kind of activity. We
would keep packets longer than necessary, which may disturb
upper layers (think: TCP). That is why we introduced the thread,
as a simple mechanism to keep packets no longer than necessary.
The locking done by this thread may create realtime issues.
The whole logic of time keeping can be changed later if it turns
out that it is not efficient.
The functions dealing with DRB AM have been removed. All the types
of PDCP entities (DRB AM, DRB UM, SRB) use the same mechanism for
transmit and receive operations. So the code has been made common
for all those types.
We don't deal with:
- entity re-establishment, entity release or entity suspend
- SDU discard (the timer is present but not used)
- status reporting
- data recovery
- data volume calculation
- header compression / decompression
There is no plan to implement those features in a near future.
Integrity protection is not done yet, and will be implemented soon.
Minimalist gnb tracer to trace (for the moment) rxdataF when some
decoding for PUCCH or PUSCH has been done.
Important: T_BUFFER_MAX and T_CACHE_SIZE have been changed. T_BUFFER_MAX
is twice as big as before. T_CACHE_SIZE has been devided by two. Globally,
the memory usage of the T tracer is the same, but since we divided by two
the number of buffers in the cache, we may hit some realtime problems if
all the buffers are busy at some point. The simple solution would be to
use 8192*2 for T_CACHE_SIZE as it was, but then we double the memory usage
of the T tracer. This, or another solution, to be defined.
- a new section in the configuration file to select security algorithms,
with new code to deal with it
- cleanup CG-ConfigInfo: specs seem to indicate that we must not
add mcg_RB_Config; the gNB has to deal with that
- as a consequence, modify fill_default_rbconfig() that is called in
every cases and needs security and bearer parameters
The new section in the configuration file looks like:
security = {
# preferred ciphering algorithms
# the first one of the list that an UE supports in chosen
# valid values: nea0, nea1, nea2, nea3
ciphering_algorithms = ( "nea0", "nea2" );
# preferred integrity algorithms
# the first one of the list that an UE supports in chosen
# valid values: nia0, nia1, nia2, nia3
integrity_algorithms = ( "nia0" );
};
The code is forced to use nea2, no matter what the UE supports.
After 2^18 PDCP packets, it will fail to work (we don't use HFN yet).
These limitations will be fixed in later commits.
The existing security function was not reused, because it does too
much memory allocation and initializes the security context at each
ciphering. So here comes nr_pdcp_security_nea2_cipher(). And also
the ciphering is done inplace. To be changed if necessary.
There was a problem with several UEs in the wireshark log. It seems that
wireshark needs the UE id otherwise it gets confused and sees retransmissions
where it's just another UE connecting and doing the same as a previous
UE.
Let's put kind of a UE id. Each RNTI seen by macpdu2wireshark gets an UE id
assigned the first time it's seen in a trace.
This is a bit hackish but seems to be fine.
We could have added UE_id in all the traces, that is change the T() calls
in the softmodem. It was less work to do it as it is, ie. only touch
macpdu2wireshark.c.
The RAR uses the same UE id all the time, may be a problem. To be refined
if needed.
The necessity of SOstart/SOend was understood after email exchanges with
David Breaux <davidbreaux67@gmail.com>.
See openair2/LAYER2/rlc_v2/tests/test46.h for some details.
Basically if we NACK without SOstart/SOend then the other end of the RLC
channel will consider all the bytes as NACKed and will have to resend them
all. If another status is then reported before all the bytes are sent,
and because we don't use SOstart/SOend this status will NACK all the bytes
again, then all the bytes are to be sent again. And the cycle repeats.
It will end with a max retx reached (which triggers RLF, Radio Link Failure).
The function put_bit in openair2/LAYER2/rlc_v2/rlc_pdu.c has been changed
too, maybe introducing some performance issues.
There was a problem with SN size. It was hardcoded to 12 bits (so max
sn = 4095) but we configure the UE with 18 bits. After a while, iperf
didn't work anymore because of this (what a surprise!).
So let's get the correct value from pdcp_Config.
And while we're at it, let's store also discard_timer and t_reordering
that will be used later by a proper NR PDCP layer (which is currently
work in progress).
RLC has timers. We need to keep track of time. We chose to not use
realtime but 'processing time'. That is we send a tick to RLC for every
subframe when it is processed by the softmodem. Before this commit we
used to increase time in the function mac_rlc_status_ind which may not be
called every subframe (especially in TDD mode). So we need another way
to keep track of time.
We introduce rlc_tick() and call it in eNB_dlsch_ulsch_scheduler, which
is called every subframe, both in FDD and TDD.
We also call rlc_tick() in the function ru_thread_synch() because
pdcp_run() and rrc_rx_tx() are called there too.
It may not be the best solution. To be revised if needed.
Note: the UE has not been touched. Some modification is needed for the UE
too, most probably.
This "error" is not a real error.
openair2/LAYER2/nr_rlc/tests/test.c includes testXX.h to define the array
int test[] and by design test[2] always exists.
Let's simply suppress the reporting of this error.
The T tracer macpdu2wireshark has been modified to handle the new traces.
The usage is the same as for LTE.
In NR the RACH is logged as an LTE RACH. Maybe we'll change that in the
future. Looking at the source code of wireshark, I could not find any
way to report NR RACH. Maybe they forgot it. Or I did not search correctly...
A test with band 79 was not working. It turns out that the computation
of the frequency from the NR-ARFCN was giving a wrong value.
After fixing band 79 as found in this commit, the frequency was correct.
The array nr_bandtable has then been reviewed fully and all the values
that were wrong have been changed. The check of correctness was done
using the website https://www.sqimway.com/nr_band.php
dedicated_DRB was not set properly when the NSA UE sends
ReconfigurationComplete so the wrong code was executed below
(we had dedicated_DRB == 0). Let's put a value that does
nothing (-1 seems fine).
After the UE disconnects, the next connection fails
because the gNB cannot decode Msg3. The root cause
seems to be this ul_valid_re_per_slot not set to 0.
Setting it before the test fixes the problem and
does not seem to impact anything else.
Some data remains in the PHY of the gNB after UE removal by
upper layers. After some connection/deconnection/reconnection
the gNB will exit because there is no free space in PHY.
This commit adds some global variables shared by MAC and PHY.
The function mac_remove_nr_ue puts UEs to be removed in it
and the function rxtx removes what needs to be removed.
This is a hack. It will be removed at some point in time.
The real way to remove in the PHY should be to release
data after some time of inactivity in the PHY, which would
mean the UE has been removed in upper layers. Kind of
timer-based thing. There should be no communication between
PHY and MAC.
This commit introduces several enhancements in X2AP and RRC to deal
with ENDC. The main goal is to deal with UE's on/off and with various
errors.
The following work is included:
- DC prep and DC overall X2 timers
- X2 messages for UE release and DC timers' timeout
Open problems:
- what to do when the UE sends an scg Failure? This commit removes
the UE. And this removal may not be correctly done
- UE removal in the gNB's PHY layer (a next commit is coming,
but it is hackish and will need a rework)
RLC has timers. We need to keep track of time. We chose to not use
realtime but 'processing time'. That is we send a tick to RLC for every
subframe when it is processed by the softmodem. Before this commit we
used to increase time in the function mac_rlc_status_ind which is called
for every subframe in the eNB, but not in the gNB (at least not in the
current gNB). So we need another way to keep track of time.
The function gNB_dlsch_ulsch_scheduler is called for every slot. It seems
to be the perfect candidate to send a tick to RLC for every subframe. We
take care of subcarrier spacing to send every subframe (one subframe is
always 1ms), not every slot (slot duration depends on subcarrier spacing).
It may not be the best solution. To be revised if needed.
Note: the UE has not been fixed. Some modification is needed for the UE
too, most probably.
c_SRS was 61, which does not fit for 106 PRBs as it seems. Value 0 works.
Maybe not the right thing to do though (see 38.211 6.4.1.4.3,
table 6.4.1.4.3-1).
Also it seems necessary to pass desired LTE/NR bands when requesting
LTE UE capabilities. Today this is hardcoded, to be done better at
some point. We should get the bands from the eNB configuration and
the eNB/gNB X2 exchanges.
c_SRS was 61, which does not fit for 106 PRBs as it seems. Value 0 works.
Maybe not the right thing to do though (see 38.211 6.4.1.4.3,
table 6.4.1.4.3-1).
Also it seems necessary to pass desired LTE/NR bands when requesting
LTE UE capabilities. Today this is hardcoded, to be done better at
some point. We should get the bands from the eNB configuration and
the eNB/gNB X2 exchanges.
When running the softmodem, we get the error:
fill_ulsch UE_id 0 nb_rb = 0
The code has changed some time ago and this log was left but is not
meaningful anymore.
hack: in schedule_ue_spec, 3 bytes are "reserved" (this should be
done better). An RLC AM entity may ask for only 2 bytes for
ACKing and get a TBS of 32 bits and due to these 3 reserved bytes we may
end up with only 1 byte left for RLC, which is not enough. This hack
prevents the problem. To be done better at some point. If the function
schedule_ue_spec is (has been) reworked, this hack can be removed.
Dig for "TBS - ta_len - header_length_total - sdu_length_total - 3"
in schedule_ue_spec.
This should be done better at some point.
Minor fixes, doesn't change anything. Not sure these are 'bugs' either,
but let's be polite with coverity scan...
One thing was not changed.
Coverity scan says:
*** CID 357991: Memory - illegal accesses (USE_AFTER_FREE)
/home/carabe/raphael/openairinterface5g/openair2/LAYER2/rlc_v2/rlc_entity_am.c: 507 in tx_list_remove_sn()
501 } else {
502 prev = cur;
503 cur = cur->next;
504 }
505 }
506
>>> CID 357991: Memory - illegal accesses (USE_AFTER_FREE)
>>> Using freed pointer "head.next".
507 return head.next;
508 }
509
510 void cleanup_sdu_list(rlc_entity_am_t *entity)
511 {
512 rlc_sdu_t head;
But as far as I understand, there is no problem. We don't access
head.next at all if it has been freed. Or is there some aliasing
going on there (pointer aliasing)? I doubt it. False positive?
Not sure if the problem is in the MAC layer or in the RLC v2 layer.
Maybe the buffer status reporting in RLC v2 is not good. Let's remove
two AssertFatal in the MAC for the time being.
Limitation: this fix only works if there is only one eNB,
because we force module_id = 0 in the function deliver_sdu where
(as the code is today) we have no way to know which module_id is
concerned. More generally, the rlc v2 does not work with more than
1 eNB.
The old method was wrong.
The new method is better, but not accurate, especially when the
TX buffer contains SDU of size > 2047 bytes.
Plus it's probably too slow.
To be refined at some point.
This commit introduces an alternative RLC implementation
based on 36.322 version 8. AM and UM done, TM not done.
See openair2/LAYER2/rlc_v2/TODO for missing parts.
Seems like the UEs in the CI don't like when asked to report NR UE
capabilities. So let's first ask for eutra capabilities and then
dig into them to check for en-dc support and only then ask for
NR capabilities. We chose to ask for NR capabilities after receiving
RRCConnectionReconfigurationComplete. That may not be the best thing
to do.
Today, we do nothing with those capabilities. We could completely skip
asking them to the UE, but since the code is there and since it may be
useful in the future, let's do it. Hopefully, it should not hurt too much.
Cedric Roux