[srslte-users] ue crash on multiple simultaneous connections.
J Giovatto
jgiovatto at adjacentlink.com
Tue Sep 28 00:12:04 UTC 2021
Ha yea, good point.
Thanks for looking into it.
Best
Joe
On 9/27/21 15:03, Andre Puschmann wrote:
> Hi Joe,
>
> the below proposed approach is "quite" ok. Problem is that, in theory,
> between emptying and resizing another SDU could have been added again.
> So keeping the lock while doing both is actually required.
>
> Anyway, we've fixed the issue internally and will provide the update
> with the next release.
>
> Thanks
> Andre
>
>
> On 27/9/21 19:28, J Giovatto wrote:
>> Hi
>>
>> Seems like this is a corner case,as a work around, I might propose
>> this as a possible solution to the queue resize failure which seems
>> to bring down the house.
>>
>>
>> diff --git a/lib/src/upper/rlc_am_lte.cc b/lib/src/upper/rlc_am_lte.cc
>> index 23a182957..457e7b921 100644
>> --- a/lib/src/upper/rlc_am_lte.cc
>> +++ b/lib/src/upper/rlc_am_lte.cc
>> @@ -319,6 +319,8 @@ bool rlc_am_lte::rlc_am_lte_tx::configure(const
>> rlc_config_t& cfg_)
>> poll_retx_timer.set(static_cast<uint32_t>(cfg.t_poll_retx),
>> [this](uint32_t timerid) { timer_expired(timerid); });
>> }
>>
>> + empty_queue();
>> +
>> tx_sdu_queue.resize(cfg_.tx_queue_length);
>>
>> tx_enabled = true;
>>
>>
>> Let me know what you think, when you get a chance.
>>
>> Joe
>>
>> On 9/25/21 02:19, Andrea Valori wrote:
>>> Joe,
>>> from my (inexperienced) empirical evidence, seems that the software
>>> is EXTREMELY demanding and when the hardware is suffering, there are
>>> errors but not a clear "resources too low" message that anyone can
>>> understand.
>>> If you don't have better suggestions, I suggest you try to use a
>>> more powerful machine and in case work your way down from there.
>>> If you have the GUI, I suggest you disable it since it will leave
>>> much more resources to the srs stack (I was amazed seeing the
>>> difference between performances with and without UI).
>>> Cheers
>>>
>>> On Fri, 24 Sept 2021 at 00:58, J Giovatto
>>> <jgiovatto at adjacentlink.com> wrote:
>>>
>>> Hi folks,
>>>
>>> Here is the crash dump for a ue.
>>>
>>> This is repeatable when 5 or more ues, try to connect
>>> simultaneously,
>>> (start 5 UE's then start the enb)
>>>
>>>
>>> Found Cell: Mode=FDD, PCI=1, PRB=50, Ports=1, CFO=0.0 KHz
>>> Found PLMN: Id=00101, TAC=7
>>> Random Access Transmission: seq=14, tti=7541, ra-rnti=0x2
>>> Random Access Transmission: seq=41, tti=7561, ra-rnti=0x2
>>> Random Access Transmission: seq=17, tti=7581, ra-rnti=0x2
>>> Random Access Transmission: seq=8, tti=7601, ra-rnti=0x2
>>> Random Access Transmission: seq=13, tti=7621, ra-rnti=0x2
>>> Random Access Transmission: seq=27, tti=7711, ra-rnti=0x2
>>> Random Access Complete. c-rnti=0x58, ta=0
>>> RRC Connected
>>> RRC Connected
>>> /home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/adt/circular_buffer.h:547:
>>> Dynamic resizes not supported when circular buffer is not empty
>>> Stack trace (most recent call last) in thread 826286:
>>> #20 Object "", at 0xffffffffffffffff, in
>>> #19 Source "../sysdeps/unix/sysv/linux/x86_64/clone.S", line
>>> 95, in
>>> __clone [0x7fdf59f294c2]
>>> #18 Source
>>> "/usr/src/debug/glibc-2.32-10.fc33.x86_64/nptl/pthread_create.c", line
>>> 463, in start_thread [0x7fdf5add93f8]
>>> 460: ret = (void*) (uintptr_t) start (pd->arg);
>>> 461: }
>>> 462: else
>>> > 463: ret = pd->start_routine (pd->arg);
>>> 464: THREAD_SETMEM (pd, result, ret);
>>> 465: }
>>> #17 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/common/threads.h",
>>> line 111, in thread_function_entry [0xc59291]
>>> 108: static void* thread_function_entry(void* _this)
>>> 109: {
>>> 110: pthread_setname_np(pthread_self(),
>>> ((thread*)_this)->name.c_str());
>>> > 111: ((thread*)_this)->run_thread();
>>> 112: return NULL;
>>> 113: }
>>> #16 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/srsue/src/stack/ue_stack_lte.cc",
>>> line 332, in run_thread [0xd0f67f]
>>> 329: void ue_stack_lte::run_thread()
>>> 330: {
>>> 331: while (running) {
>>> > 332: task_sched.run_next_task();
>>> 333: }
>>> 334: }
>>> #15 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/common/task_scheduler.h",
>>> line 81, in run_next_task [0xd12754]
>>> 78: srsran::move_task_t task{};
>>> 79: if (external_tasks.wait_pop(&task) >= 0) {
>>> 80: task();
>>> > 81: run_all_internal_tasks();
>>> 82: return true;
>>> 83: }
>>> 84: run_all_internal_tasks();
>>> #14 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/common/task_scheduler.h",
>>> line 108, in run_all_internal_tasks [0xd127f0]
>>> 105: // Note: Using a deque because tasks can
>>> enqueue new
>>> tasks, which would lead to
>>> 106: // reference invalidation in case of vector
>>> 107: while (not internal_tasks.empty()) {
>>> > 108: internal_tasks.front()();
>>> 109: internal_tasks.pop_front();
>>> 110: }
>>> 111: }
>>> #13 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/adt/move_callback.h",
>>> line 178, in operator() [0xd1448d]
>>> 175: return *this;
>>> 176: }
>>> 177:
>>> > 178: R operator()(Args... args) const noexcept { return
>>> oper_ptr->call(&buffer, std::forward<Args>(args)...); }
>>> 179:
>>> 180: bool is_empty() const { return oper_ptr ==
>>> &empty_table; }
>>> 181: bool is_in_small_buffer() const { return
>>> oper_ptr->is_in_small_buffer(); }
>>> #12 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/adt/move_callback.h",
>>> line 101, in call [0xd8efdc]
>>> 98: {
>>> 99: public:
>>> 100: constexpr heap_table_t() = default;
>>> > 101: R call(void* src, Args... args) const final {
>>> return
>>> (**static_cast<FunT**>(src))(std::forward<Args>(args)...); }
>>> 102: void move(void* src, void* dest) const final
>>> 103: {
>>> 104: *static_cast<FunT**>(dest) =
>>> *static_cast<FunT**>(src);
>>> #11 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/srsue/src/stack/rrc/rrc.cc",
>>>
>>> line 1675, in operator() [0xd885a1]
>>> 1672: case
>>> dl_ccch_msg_type_c::c1_c_::types::rrc_conn_setup: {
>>> 1673: transaction_id =
>>> c1->rrc_conn_setup().rrc_transaction_id;
>>> 1674: rrc_conn_setup_s conn_setup_copy =
>>> c1->rrc_conn_setup();
>>> >1675: task_sched.defer_task([this,
>>> conn_setup_copy]() {
>>> handle_con_setup(conn_setup_copy); });
>>> 1676: break;
>>> 1677: }
>>> 1678: case
>>> dl_ccch_msg_type_c::c1_c_::types::rrc_conn_reest: {
>>> #10 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/srsue/src/stack/rrc/rrc.cc",
>>>
>>> line 2543, in handle_con_setup [0xd8c559]
>>> 2540: srsran::console("RRC Connected\n");
>>> 2541:
>>> 2542: // Apply the Radio Resource configuration
>>> >2543:
>>> apply_rr_config_dedicated(&setup.crit_exts.c1().rrc_conn_setup_r8().rr_cfg_ded);
>>> 2544:
>>> 2545: nas->set_barring(srsran::barring_t::none);
>>> #9 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/srsue/src/stack/rrc/rrc.cc",
>>>
>>> line 2422, in apply_rr_config_dedicated [0xd8bff0]
>>> 2419: }
>>> 2420: for (uint32_t i = 0; i <
>>> cnfg->srb_to_add_mod_list.size(); i++) {
>>> 2421: // TODO: handle SRB modification
>>> >2422: add_srb(cnfg->srb_to_add_mod_list[i]);
>>> 2423: }
>>> 2424: for (uint32_t i = 0; i <
>>> cnfg->drb_to_release_list.size(); i++) {
>>> 2425: release_drb(cnfg->drb_to_release_list[i]);
>>> #8 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/srsue/src/stack/rrc/rrc.cc",
>>>
>>> line 2572, in add_srb [0xd8c85a]
>>> 2570: // Setup RLC
>>> 2571: if (srb_cnfg.rlc_cfg_present) {
>>> >2572: rlc->add_bearer(srb_cnfg.srb_id,
>>> make_rlc_config_t(srb_cnfg));
>>> 2573: }
>>> 2574:
>>> 2575: // Setup MAC
>>> #7 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/src/upper/rlc.cc",
>>> line
>>> 480, in add_bearer [0x106d1c4]
>>> 478: // configure and add to array
>>> 479: if (cnfg.rlc_mode != rlc_mode_t::tm and
>>> rlc_array.find(lcid) != rlc_array.end()) {
>>> > 480: if (not rlc_array.at
>>> <http://rlc_array.at>(lcid)->configure(cnfg)) {
>>> 481: logger.error("Error configuring RLC entity.");
>>> 482: goto delete_and_exit;
>>> 483: }
>>> #6 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/src/upper/rlc_am_lte.cc",
>>> line 159, in configure [0x1086329]
>>> 156: return false;
>>> 157: }
>>> 158:
>>> > 159: if (not tx.configure(cfg)) {
>>> 160: logger.error("Error configuring bearer (TX)");
>>> 161: return false;
>>> 162: }
>>> #5 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/src/upper/rlc_am_lte.cc",
>>> line 322, in configure [0x1086eaa]
>>> 319:
>>> poll_retx_timer.set(static_cast<uint32_t>(cfg.t_poll_retx),
>>> [this](uint32_t timerid) { timer_expired(timerid); });
>>> 320: }
>>> 321:
>>> > 322: tx_sdu_queue.resize(cfg_.tx_queue_length);
>>> 323:
>>> 324: tx_enabled = true;
>>> #4 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/upper/byte_buffer_queue.h",
>>> line 60, in resize [0x107be12]
>>> 58: bool try_read(unique_byte_buffer_t* msg) { return
>>> queue.try_pop(*msg); }
>>> 59:
>>> > 60: void resize(uint32_t capacity) {
>>> queue.set_size(capacity); }
>>> 61: uint32_t size() { return (uint32_t)queue.size(); }
>>> 62: uint32_t get_n_sdus() { return n_sdus; }
>>> #3 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/adt/circular_buffer.h",
>>> line 601, in set_size [0x107bfb8]
>>> 598: explicit dyn_blocking_queue(size_t size,
>>> PushingCallback
>>> push_callback = {}, PoppingCallback pop_callback = {}) :
>>> 599: base_t(push_callback, pop_callback, size)
>>> 600: {}
>>> > 601: void set_size(size_t size) {
>>> base_t::circ_buffer.set_size(size); }
>>> 602:
>>> 603: template <typename F>
>>> 604: bool apply_first(const F& func)
>>> #2 Source
>>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/adt/circular_buffer.h",
>>> line 547, in set_size [0x107f732]
>>> 545: void set_size(size_t sz)
>>> 546: {
>>> > 547: srsran_assert(base_t::empty() or sz ==
>>> base_t::size(),
>>> 548: "Dynamic resizes not supported when
>>> circular buffer is not empty");
>>> 549: base_t::buffer.resize(sz);
>>> 550: }
>>> #1 Source
>>> "/usr/src/debug/glibc-2.32-10.fc33.x86_64/stdlib/abort.c",
>>> line 79, in abort [0x7fdf59e4e8a3]
>>> 76: stage = 0;
>>> 77: __libc_lock_unlock_recursive (lock);
>>> 78:
>>> > 79: raise (SIGABRT);
>>> 80:
>>> 81: __libc_lock_lock_recursive (lock);
>>> 82: stage = save_stage + 1;
>>> #0 Source "../sysdeps/unix/sysv/linux/raise.c", line 50, in
>>> raise
>>> [0x7fdf59e659e5]
>>> Aborted (Signal sent by tkill() 826270 0)
>>> Aborted (core dumped)
>>>
>>> Best
>>>
>>> Joe
>>>
>>> _______________________________________________
>>> srslte-users mailing list
>>> srslte-users at lists.softwareradiosystems.com
>>> https://lists.softwareradiosystems.com/mailman/listinfo/srslte-users
>>>
>>
>>
>> _______________________________________________
>> srslte-users mailing list
>> srslte-users at lists.softwareradiosystems.com
>> https://lists.softwareradiosystems.com/mailman/listinfo/srslte-users
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://lists.srsran.com/pipermail/srsran-users/attachments/20210927/68aacb27/attachment.sig>
More information about the srsran-users
mailing list