[srslte-users] ue crash on multiple simultaneous connections.
Andre Puschmann
andre at srs.io
Mon Sep 27 19:03:56 UTC 2021
Hi Joe,
the below proposed approach is "quite" ok. Problem is that, in theory,
between emptying and resizing another SDU could have been added again.
So keeping the lock while doing both is actually required.
Anyway, we've fixed the issue internally and will provide the update
with the next release.
Thanks
Andre
On 27/9/21 19:28, J Giovatto wrote:
> Hi
>
> Seems like this is a corner case,as a work around, I might propose this
> as a possible solution to the queue resize failure which seems to bring
> down the house.
>
>
> diff --git a/lib/src/upper/rlc_am_lte.cc b/lib/src/upper/rlc_am_lte.cc
> index 23a182957..457e7b921 100644
> --- a/lib/src/upper/rlc_am_lte.cc
> +++ b/lib/src/upper/rlc_am_lte.cc
> @@ -319,6 +319,8 @@ bool rlc_am_lte::rlc_am_lte_tx::configure(const
> rlc_config_t& cfg_)
> poll_retx_timer.set(static_cast<uint32_t>(cfg.t_poll_retx),
> [this](uint32_t timerid) { timer_expired(timerid); });
> }
>
> + empty_queue();
> +
> tx_sdu_queue.resize(cfg_.tx_queue_length);
>
> tx_enabled = true;
>
>
> Let me know what you think, when you get a chance.
>
> Joe
>
> On 9/25/21 02:19, Andrea Valori wrote:
>> Joe,
>> from my (inexperienced) empirical evidence, seems that the software is
>> EXTREMELY demanding and when the hardware is suffering, there are
>> errors but not a clear "resources too low" message that anyone can
>> understand.
>> If you don't have better suggestions, I suggest you try to use a more
>> powerful machine and in case work your way down from there.
>> If you have the GUI, I suggest you disable it since it will leave much
>> more resources to the srs stack (I was amazed seeing the difference
>> between performances with and without UI).
>> Cheers
>>
>> On Fri, 24 Sept 2021 at 00:58, J Giovatto <jgiovatto at adjacentlink.com>
>> wrote:
>>
>> Hi folks,
>>
>> Here is the crash dump for a ue.
>>
>> This is repeatable when 5 or more ues, try to connect simultaneously,
>> (start 5 UE's then start the enb)
>>
>>
>> Found Cell: Mode=FDD, PCI=1, PRB=50, Ports=1, CFO=0.0 KHz
>> Found PLMN: Id=00101, TAC=7
>> Random Access Transmission: seq=14, tti=7541, ra-rnti=0x2
>> Random Access Transmission: seq=41, tti=7561, ra-rnti=0x2
>> Random Access Transmission: seq=17, tti=7581, ra-rnti=0x2
>> Random Access Transmission: seq=8, tti=7601, ra-rnti=0x2
>> Random Access Transmission: seq=13, tti=7621, ra-rnti=0x2
>> Random Access Transmission: seq=27, tti=7711, ra-rnti=0x2
>> Random Access Complete. c-rnti=0x58, ta=0
>> RRC Connected
>> RRC Connected
>> /home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/adt/circular_buffer.h:547:
>> Dynamic resizes not supported when circular buffer is not empty
>> Stack trace (most recent call last) in thread 826286:
>> #20 Object "", at 0xffffffffffffffff, in
>> #19 Source "../sysdeps/unix/sysv/linux/x86_64/clone.S", line 95, in
>> __clone [0x7fdf59f294c2]
>> #18 Source
>> "/usr/src/debug/glibc-2.32-10.fc33.x86_64/nptl/pthread_create.c", line
>> 463, in start_thread [0x7fdf5add93f8]
>> 460: ret = (void*) (uintptr_t) start (pd->arg);
>> 461: }
>> 462: else
>> > 463: ret = pd->start_routine (pd->arg);
>> 464: THREAD_SETMEM (pd, result, ret);
>> 465: }
>> #17 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/common/threads.h",
>> line 111, in thread_function_entry [0xc59291]
>> 108: static void* thread_function_entry(void* _this)
>> 109: {
>> 110: pthread_setname_np(pthread_self(),
>> ((thread*)_this)->name.c_str());
>> > 111: ((thread*)_this)->run_thread();
>> 112: return NULL;
>> 113: }
>> #16 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/srsue/src/stack/ue_stack_lte.cc",
>> line 332, in run_thread [0xd0f67f]
>> 329: void ue_stack_lte::run_thread()
>> 330: {
>> 331: while (running) {
>> > 332: task_sched.run_next_task();
>> 333: }
>> 334: }
>> #15 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/common/task_scheduler.h",
>> line 81, in run_next_task [0xd12754]
>> 78: srsran::move_task_t task{};
>> 79: if (external_tasks.wait_pop(&task) >= 0) {
>> 80: task();
>> > 81: run_all_internal_tasks();
>> 82: return true;
>> 83: }
>> 84: run_all_internal_tasks();
>> #14 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/common/task_scheduler.h",
>> line 108, in run_all_internal_tasks [0xd127f0]
>> 105: // Note: Using a deque because tasks can enqueue new
>> tasks, which would lead to
>> 106: // reference invalidation in case of vector
>> 107: while (not internal_tasks.empty()) {
>> > 108: internal_tasks.front()();
>> 109: internal_tasks.pop_front();
>> 110: }
>> 111: }
>> #13 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/adt/move_callback.h",
>> line 178, in operator() [0xd1448d]
>> 175: return *this;
>> 176: }
>> 177:
>> > 178: R operator()(Args... args) const noexcept { return
>> oper_ptr->call(&buffer, std::forward<Args>(args)...); }
>> 179:
>> 180: bool is_empty() const { return oper_ptr ==
>> &empty_table; }
>> 181: bool is_in_small_buffer() const { return
>> oper_ptr->is_in_small_buffer(); }
>> #12 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/adt/move_callback.h",
>> line 101, in call [0xd8efdc]
>> 98: {
>> 99: public:
>> 100: constexpr heap_table_t() = default;
>> > 101: R call(void* src, Args... args) const final { return
>> (**static_cast<FunT**>(src))(std::forward<Args>(args)...); }
>> 102: void move(void* src, void* dest) const final
>> 103: {
>> 104: *static_cast<FunT**>(dest) =
>> *static_cast<FunT**>(src);
>> #11 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/srsue/src/stack/rrc/rrc.cc",
>> line 1675, in operator() [0xd885a1]
>> 1672: case
>> dl_ccch_msg_type_c::c1_c_::types::rrc_conn_setup: {
>> 1673: transaction_id =
>> c1->rrc_conn_setup().rrc_transaction_id;
>> 1674: rrc_conn_setup_s conn_setup_copy =
>> c1->rrc_conn_setup();
>> >1675: task_sched.defer_task([this, conn_setup_copy]() {
>> handle_con_setup(conn_setup_copy); });
>> 1676: break;
>> 1677: }
>> 1678: case
>> dl_ccch_msg_type_c::c1_c_::types::rrc_conn_reest: {
>> #10 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/srsue/src/stack/rrc/rrc.cc",
>> line 2543, in handle_con_setup [0xd8c559]
>> 2540: srsran::console("RRC Connected\n");
>> 2541:
>> 2542: // Apply the Radio Resource configuration
>> >2543:
>> apply_rr_config_dedicated(&setup.crit_exts.c1().rrc_conn_setup_r8().rr_cfg_ded);
>> 2544:
>> 2545: nas->set_barring(srsran::barring_t::none);
>> #9 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/srsue/src/stack/rrc/rrc.cc",
>> line 2422, in apply_rr_config_dedicated [0xd8bff0]
>> 2419: }
>> 2420: for (uint32_t i = 0; i <
>> cnfg->srb_to_add_mod_list.size(); i++) {
>> 2421: // TODO: handle SRB modification
>> >2422: add_srb(cnfg->srb_to_add_mod_list[i]);
>> 2423: }
>> 2424: for (uint32_t i = 0; i <
>> cnfg->drb_to_release_list.size(); i++) {
>> 2425: release_drb(cnfg->drb_to_release_list[i]);
>> #8 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/srsue/src/stack/rrc/rrc.cc",
>> line 2572, in add_srb [0xd8c85a]
>> 2570: // Setup RLC
>> 2571: if (srb_cnfg.rlc_cfg_present) {
>> >2572: rlc->add_bearer(srb_cnfg.srb_id,
>> make_rlc_config_t(srb_cnfg));
>> 2573: }
>> 2574:
>> 2575: // Setup MAC
>> #7 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/src/upper/rlc.cc",
>> line
>> 480, in add_bearer [0x106d1c4]
>> 478: // configure and add to array
>> 479: if (cnfg.rlc_mode != rlc_mode_t::tm and
>> rlc_array.find(lcid) != rlc_array.end()) {
>> > 480: if (not rlc_array.at
>> <http://rlc_array.at>(lcid)->configure(cnfg)) {
>> 481: logger.error("Error configuring RLC entity.");
>> 482: goto delete_and_exit;
>> 483: }
>> #6 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/src/upper/rlc_am_lte.cc",
>> line 159, in configure [0x1086329]
>> 156: return false;
>> 157: }
>> 158:
>> > 159: if (not tx.configure(cfg)) {
>> 160: logger.error("Error configuring bearer (TX)");
>> 161: return false;
>> 162: }
>> #5 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/src/upper/rlc_am_lte.cc",
>> line 322, in configure [0x1086eaa]
>> 319:
>> poll_retx_timer.set(static_cast<uint32_t>(cfg.t_poll_retx),
>> [this](uint32_t timerid) { timer_expired(timerid); });
>> 320: }
>> 321:
>> > 322: tx_sdu_queue.resize(cfg_.tx_queue_length);
>> 323:
>> 324: tx_enabled = true;
>> #4 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/upper/byte_buffer_queue.h",
>> line 60, in resize [0x107be12]
>> 58: bool try_read(unique_byte_buffer_t* msg) { return
>> queue.try_pop(*msg); }
>> 59:
>> > 60: void resize(uint32_t capacity) {
>> queue.set_size(capacity); }
>> 61: uint32_t size() { return (uint32_t)queue.size(); }
>> 62: uint32_t get_n_sdus() { return n_sdus; }
>> #3 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/adt/circular_buffer.h",
>> line 601, in set_size [0x107bfb8]
>> 598: explicit dyn_blocking_queue(size_t size,
>> PushingCallback
>> push_callback = {}, PoppingCallback pop_callback = {}) :
>> 599: base_t(push_callback, pop_callback, size)
>> 600: {}
>> > 601: void set_size(size_t size) {
>> base_t::circ_buffer.set_size(size); }
>> 602:
>> 603: template <typename F>
>> 604: bool apply_first(const F& func)
>> #2 Source
>> "/home/jgiovatto/Devel/LTE/srs/srsRAN-emane/lib/include/srsran/adt/circular_buffer.h",
>> line 547, in set_size [0x107f732]
>> 545: void set_size(size_t sz)
>> 546: {
>> > 547: srsran_assert(base_t::empty() or sz ==
>> base_t::size(),
>> 548: "Dynamic resizes not supported when
>> circular buffer is not empty");
>> 549: base_t::buffer.resize(sz);
>> 550: }
>> #1 Source
>> "/usr/src/debug/glibc-2.32-10.fc33.x86_64/stdlib/abort.c",
>> line 79, in abort [0x7fdf59e4e8a3]
>> 76: stage = 0;
>> 77: __libc_lock_unlock_recursive (lock);
>> 78:
>> > 79: raise (SIGABRT);
>> 80:
>> 81: __libc_lock_lock_recursive (lock);
>> 82: stage = save_stage + 1;
>> #0 Source "../sysdeps/unix/sysv/linux/raise.c", line 50, in raise
>> [0x7fdf59e659e5]
>> Aborted (Signal sent by tkill() 826270 0)
>> Aborted (core dumped)
>>
>> Best
>>
>> Joe
>>
>> _______________________________________________
>> srslte-users mailing list
>> srslte-users at lists.softwareradiosystems.com
>> https://lists.softwareradiosystems.com/mailman/listinfo/srslte-users
>>
>
>
> _______________________________________________
> srslte-users mailing list
> srslte-users at lists.softwareradiosystems.com
> https://lists.softwareradiosystems.com/mailman/listinfo/srslte-users
>
More information about the srsran-users
mailing list