slam/basalt: segfault in ui_runner()
Have had the following segfault with a recent Monado + monado-basalt built from the sources twice, within a few minutes. Initially monado-service ran fine, but seems to have crashed upon strong head movements.
$ env XRT_DEBUG_GUI=on SLAM_UI=on XRT_COMPOSITOR_COMPUTE=true SURVIVE_TIMECODE_OFFSET_MS=-6.94 ~/.local/bin/monado-service
...
WARN [comp_renderer_draw] Compositor probably missed frame by 11.16ms
WARN [wait_for_scheduled_free] 132610195.928ms: Dropping old missed frame in favour for completed new frame
WARN [comp_renderer_draw] Compositor probably missed frame by 11.29ms
WARN [wait_for_scheduled_free] 132610229.402ms: Dropping old missed frame in favour for completed new frame
WARN [comp_renderer_draw] Compositor probably missed frame by 11.20ms
WARN [wait_for_scheduled_free] 132610262.673ms: Dropping old missed frame in favour for completed new frame
WARN [comp_renderer_draw] Compositor probably missed frame by 11.09ms
WARN [wait_for_scheduled_free] 132610362.151ms: Dropping old missed frame in favour for completed new frame
WARN [comp_renderer_draw] Compositor probably missed frame by 11.28ms
WARN [wait_for_scheduled_free] 132610429.912ms: Dropping old missed frame in favour for completed new frame
WARN [comp_renderer_draw] Compositor probably missed frame by 11.14ms
WARN [wait_for_scheduled_free] 132611751.844ms: Dropping old missed frame in favour for completed new frame
zsh: segmentation fault (core dumped) env XRT_DEBUG_GUI=on SLAM_UI=on XRT_COMPOSITOR_COMPUTE=true
$ sudo coredumpctl dump > /tmp/monado-service-core.dump
[sudo] password for linus:
PID: 125926 (monado-service)
UID: 1000 (linus)
GID: 1000 (linus)
Signal: 11 (SEGV)
Timestamp: Tue 2023-05-16 17:05:54 CEST (4min 40s ago)
Command Line: /home/linus/.local/bin/monado-service
Executable: /home/linus/.local/bin/monado-service
Control Group: /user.slice/user-1000.slice/session-114.scope
Unit: session-114.scope
Slice: user-1000.slice
Session: 114
Owner UID: 1000 (linus)
Boot ID: df567cbf08624b87a3f3e95bc0e314b8
Machine ID: c7caee4282714fb09cf533de05b81b41
Hostname: linus-lptp
Storage: /var/lib/systemd/coredump/core.monado-service.1000.df567cbf08624b87a3f3e95bc0e314b8.125926.1684249554000000.zst (present)
Size on Disk: 43.8M
Package: systemd/252.6-1
build-id: 45560a6eca5e8e76f48ccae3788d037d7767514b
Message: Process 125926 (monado-service) of user 1000 dumped core.
Module /home/linus/.local/bin/monado-service from deb systemd-252.6-1.amd64
Module /home/linus/dev-priv/vr/basalt-install/lib/libbasalt.so from deb systemd-252.6-1.amd64
Module libudev.so.1 from deb systemd-252.6-1.amd64
Module libsystemd.so.0 from deb systemd-252.6-1.amd64
Stack trace of thread 125964:
#0 0x00007fcc904b78ad n/a (/home/linus/dev-priv/vr/basalt-install/lib/libbasalt.so + 0xb78ad)
#1 0x00007fcc8f0d44a3 n/a (libstdc++.so.6 + 0xd44a3)
#2 0x00007fcc8f0d44a3 n/a (libstdc++.so.6 + 0xd44a3)
#3 0x00007fcc8eea7fd4 n/a (libc.so.6 + 0x88fd4)
#4 0x00007fcc8ef285bc n/a (libc.so.6 + 0x1095bc)
Stack trace of thread 125931:
#0 0x00007fcc8eea4d36 n/a (libc.so.6 + 0x85d36)
#1 0x00007fcc8eea73f8 pthread_cond_wait (libc.so.6 + 0x883f8)
#2 0x00007fcc8628c8ab n/a (libopenblas.so.0 + 0x3748ab)
#3 0x00007fcc8eea7fd4 n/a (libc.so.6 + 0x88fd4)
#4 0x00007fcc8ef285bc n/a (libc.so.6 + 0x1095bc)
Stack trace of thread 125933:
#0 0x00007fcc8eea4d36 n/a (libc.so.6 + 0x85d36)
#1 0x00007fcc8eea73f8 pthread_cond_wait (libc.so.6 + 0x883f8)
#2 0x00007fcc8628c8ab n/a (libopenblas.so.0 + 0x3748ab)
#3 0x00007fcc8eea7fd4 n/a (libc.so.6 + 0x88fd4)
#4 0x00007fcc8ef285bc n/a (libc.so.6 + 0x1095bc)
Stack trace of thread 125932:
#0 0x00007fcc8eea4d36 n/a (libc.so.6 + 0x85d36)
#1 0x00007fcc8eea73f8 pthread_cond_wait (libc.so.6 + 0x883f8)
#2 0x00007fcc8628c8ab n/a (libopenblas.so.0 + 0x3748ab)
#3 0x00007fcc8eea7fd4 n/a (libc.so.6 + 0x88fd4)
#4 0x00007fcc8ef285bc n/a (libc.so.6 + 0x1095bc)
Stack trace of thread 125951:
#0 0x00007fcc90493816 n/a (/home/linus/dev-priv/vr/basalt-install/lib/libbasalt.so + 0x93816)
#1 0x0000556259f8a78c n/a (/home/linus/.local/bin/monado-service + 0x36e78c)
#2 0x0000556259f8a93a n/a (/home/linus/.local/bin/monado-service + 0x36e93a)
#3 0x0000556259e1ea62 n/a (/home/linus/.local/bin/monado-service + 0x202a62)
#4 0x0000556259e1eefd n/a (/home/linus/.local/bin/monado-service + 0x202efd)
#5 0x0000556259e205a7 n/a (/home/linus/.local/bin/monado-service + 0x2045a7)
#6 0x0000556259e21326 n/a (/home/linus/.local/bin/monado-service + 0x205326)
#7 0x00007fcc90fe1e95 n/a (libusb-1.0.so.0 + 0xbe95)
#8 0x00007fcc90fe82f4 n/a (libusb-1.0.so.0 + 0x122f4)
#9 0x00007fcc90fe90a2 n/a (libusb-1.0.so.0 + 0x130a2)
#10 0x00007fcc90fe183a n/a (libusb-1.0.so.0 + 0xb83a)
#11 0x00007fcc90fe2d78 libusb_handle_events_timeout_completed (libusb-1.0.so.0 + 0xcd78)
#12 0x00007fcc90fe2e30 libusb_handle_events_completed (libusb-1.0.so.0 + 0xce30)
#13 0x0000556259e20b49 n/a (/home/linus/.local/bin/monado-service + 0x204b49)
#14 0x00007fcc8eea7fd4 n/a (libc.so.6 + 0x88fd4)
#15 0x00007fcc8ef285bc n/a (libc.so.6 + 0x1095bc)
Stack trace of thread 125926:
#0 0x00007fcc8eeee345 clock_nanosleep (libc.so.6 + 0xcf345)
#1 0x00007fcc8eef2c53 __nanosleep (libc.so.6 + 0xd3c53)
#2 0x0000556259c4639b n/a (/home/linus/.local/bin/monado-service + 0x2a39b)
#3 0x0000556259c47987 n/a (/home/linus/.local/bin/monado-service + 0x2b987)
#4 0x0000556259c48116 n/a (/home/linus/.local/bin/monado-service + 0x2c116)
#5 0x0000556259c445f3 n/a (/home/linus/.local/bin/monado-service + 0x285f3)
#6 0x00007fcc8ee4618a n/a (libc.so.6 + 0x2718a)
#7 0x00007fcc8ee46245 __libc_start_main (libc.so.6 + 0x27245)
#8 0x0000556259c44501 n/a (/home/linus/.local/bin/monado-service + 0x28501)
ELF object binary architecture: AMD x86-64
More than one entry matches, ignoring rest.
gdb ~/.local/bin/monado-service /tmp/monado-service-core.dump <─(Tue,May16)─┘
GNU gdb (Debian 13.1-2) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/linus/.local/bin/monado-service...
warning: Can't open file /memfd:pulseaudio (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/monado_shm (deleted) during file-backed mapping note processing
warning: Can't open file /memfd:xshmfence (deleted) during file-backed mapping note processing
[New LWP 125964]
[New LWP 125931]
[New LWP 125933]
[New LWP 125932]
[New LWP 125951]
[New LWP 125926]
[New LWP 125936]
[New LWP 125927]
[New LWP 125935]
[New LWP 125928]
[New LWP 125929]
[New LWP 125937]
[New LWP 125934]
[New LWP 125930]
[New LWP 125938]
[New LWP 126030]
[New LWP 125940]
[New LWP 126003]
[New LWP 125941]
[New LWP 125969]
--Type <RET> for more, q to quit, c to continue without paging--c
[New LWP 126005]
[New LWP 125992]
[New LWP 125971]
[New LWP 125952]
[New LWP 125939]
[New LWP 125957]
[New LWP 125954]
[New LWP 125977]
[New LWP 125956]
[New LWP 125958]
[New LWP 125959]
[New LWP 125955]
[New LWP 125995]
[New LWP 126052]
[New LWP 126007]
[New LWP 125979]
[New LWP 125942]
[New LWP 125996]
[New LWP 125960]
[New LWP 125989]
[New LWP 125975]
[New LWP 125985]
[New LWP 126001]
[New LWP 125987]
[New LWP 126004]
[New LWP 125981]
[New LWP 125988]
[New LWP 125968]
[New LWP 125983]
[New LWP 125962]
[New LWP 125970]
[New LWP 126028]
[New LWP 125976]
[New LWP 125965]
[New LWP 126029]
[New LWP 126050]
[New LWP 125953]
[New LWP 126027]
[New LWP 125980]
[New LWP 125986]
[New LWP 125984]
[New LWP 125999]
[New LWP 126032]
[New LWP 126077]
[New LWP 126051]
[New LWP 125972]
[New LWP 126033]
[New LWP 125967]
[New LWP 126010]
[New LWP 126000]
[New LWP 125982]
[New LWP 125993]
[New LWP 125997]
[New LWP 125978]
[New LWP 126053]
[New LWP 126056]
[New LWP 126049]
[New LWP 126002]
[New LWP 125991]
[New LWP 126031]
[New LWP 125963]
[New LWP 125966]
[New LWP 125990]
[New LWP 125998]
[New LWP 125994]
[New LWP 126006]
[New LWP 126054]
[New LWP 126071]
[New LWP 126055]
warning: Section `.reg-xstate/125964' in core file too small.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/linus/.local/bin/monado-service'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Section `.reg-xstate/125964' in core file too small.
#0 0x00007fcc904b78ad in std::__shared_ptr<basalt::ManagedImage<unsigned short, std::allocator<unsigned short> >, (__gnu_cxx::_Lock_policy)2>::operator bool (this=0x18) at /usr/include/c++/12/bits/shared_ptr_base.h:1669
1669 explicit operator bool() const noexcept
[Current thread is 1 (Thread 0x7fcbf5ec56c0 (LWP 125964))]
(gdb) l
1664 element_type*
1665 get() const noexcept
1666 { return _M_ptr; }
1667
1668 /// Return true if the stored pointer is not null.
1669 explicit operator bool() const noexcept
1670 { return _M_ptr != nullptr; }
1671
1672 /// Return true if use_count() == 1.
1673 bool
(gdb) bt
#0 0x00007fcc904b78ad in std::__shared_ptr<basalt::ManagedImage<unsigned short, std::allocator<unsigned short> >, (__gnu_cxx::_Lock_policy)2>::operator bool (this=0x18) at /usr/include/c++/12/bits/shared_ptr_base.h:1669
#1 xrt::auxiliary::tracking::slam::slam_tracker_ui::ui_runner (this=0x55625c8d2780, T_w_i_init=..., cal=..., conf=...)
at /home/linus/dev-priv/vr/basalt-deps/basalt/src/monado/slam_tracker_ui.hpp:224
#2 0x00007fcc8f0d44a3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007fcc8eea7fd4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fcc8ef285bc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) l /home/linus/dev-priv/vr/basalt-deps/basalt/src/monado/slam_tracker_ui.hpp:224
219
220 if (curr_vis_data && curr_vis_data->opt_flow_res && curr_vis_data->opt_flow_res->input_images) {
221 auto &img_data = curr_vis_data->opt_flow_res->input_images->img_data;
222
223 for (size_t cam_id = 0; cam_id < num_cams; cam_id++) {
224 if (img_data[cam_id].img) {
225 img_view[cam_id]->SetImage(img_data[cam_id].img->ptr, img_data[cam_id].img->w, img_data[cam_id].img->h,
226 img_data[cam_id].img->pitch, fmt);
227 }
228 }
(gdb) info locals
No locals.
(gdb) bt full
#0 0x00007fcc904b78ad in std::__shared_ptr<basalt::ManagedImage<unsigned short, std::allocator<unsigned short> >, (__gnu_cxx::_Lock_policy)2>::operator bool (this=0x18) at /usr/include/c++/12/bits/shared_ptr_base.h:1669
No locals.
#1 xrt::auxiliary::tracking::slam::slam_tracker_ui::ui_runner (this=0x55625c8d2780, T_w_i_init=..., cal=..., conf=...)
at /home/linus/dev-priv/vr/basalt-deps/basalt/src/monado/slam_tracker_ui.hpp:224
cam_id = 1
img_data = <optimized out>
fmt = {glformat = 6409, gltype = 5123, scalable_internal_format = 32834}
UI_WIDTH = 200
window_name = {static npos = 18446744073709551615,
_M_dataplus = {<std::allocator<char>> = {<std::__new_allocator<char>> = {<No data fields>}, <No data fields>},
_M_p = 0x7fcbe8000fa0 "Basalt 6DoF Tracker for Monado"}, _M_string_length = 30, {
_M_local_buf = "\036", '\000' <repeats 14 times>, _M_allocated_capacity = 30}}
img_view_display = <optimized out>
plot_display = <optimized out>
img_view = {<std::_Vector_base<std::shared_ptr<pangolin::ImageView>, std::allocator<std::shared_ptr<pangolin::ImageView> > >> = {
_M_impl = {<std::allocator<std::shared_ptr<pangolin::ImageView> >> = {<std::__new_allocator<std::shared_ptr<pangolin::ImageView> >> = {<No data fields>}, <No data fields>}, <std::_Vector_base<std::shared_ptr<pangolin::ImageView>, std::allocator<std::shared_ptr<pangolin::ImageView> > >::_Vector_impl_data> = {_M_start = 0x7fcbe89d4620, _M_finish = 0x7fcbe89d4660,
_M_end_of_storage = 0x7fcbe89d4660}, <No data fields>}}, <No data fields>}
cam_p = {<Eigen::PlainObjectBase<Eigen::Matrix<double, 3, 1, 0, 3, 1> >> = {<Eigen::MatrixBase<Eigen::Matrix<double, 3, 1, 0, 3, 1> >> = {<Eigen::DenseBase<Eigen::Matrix<double, 3, 1, 0, 3, 1> >> = {<Eigen::DenseCoeffsBase<Eigen::Matrix<double, 3, 1, 0, 3, 1>, 3>> = {<Eigen::DenseCoeffsBase<Eigen::Matrix<double, 3, 1, 0, 3, 1>, 1>> = {<Eigen::DenseCoeffsBase<Eigen::Matrix<double, 3, 1, 0, 3, 1>, 0>> = {<Eigen::EigenBase<Eigen::Matrix<double, 3, 1, 0, 3, 1> >> = {<No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, m_storage = {m_data = {array = {0.71803169431605696, -2.4596260225857867,
1}}}}, <No data fields>}
camera = {modelview = {m = {0.95993275150265389, -0.10188292752563866, 0.26105359923078925, 0, 0.28023046335568885,
0.34900116775204409, -0.89424217766506942, 0, 0, 0.93156752518886943, 0.3635683512264028, 0, -4.1759139771997674e-07,
1.620956049900832e-06, -2.7505145899416394, 1}},
projection = {<std::_Vector_base<pangolin::OpenGlMatrix, std::allocator<pangolin::OpenGlMatrix> >> = {
_M_impl = {<std::allocator<pangolin::OpenGlMatrix>> = {<std::__new_allocator<pangolin::OpenGlMatrix>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<pangolin::OpenGlMatrix, std::allocator<pangolin::OpenGlMatrix> >::_Vector_impl_data> = {
_M_start = 0x7fcbe89c5e20, _M_finish = 0x7fcbe89c5ea0,
_M_end_of_storage = 0x7fcbe89c5ea0}, <No data fields>}}, <No data fields>},
modelview_premult = {<std::_Vector_base<pangolin::OpenGlMatrix, std::allocator<pangolin::OpenGlMatrix> >> = {
--Type <RET> for more, q to quit, c to continue without paging--c
_M_impl = {<std::allocator<pangolin::OpenGlMatrix>> = {<std::__new_allocator<pangolin::OpenGlMatrix>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<pangolin::OpenGlMatrix, std::allocator<pangolin::OpenGlMatrix> >::_Vector_impl_data> = {
_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, <No data fields>}}, <No data fields>}, T_cw = {m = {1,
0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0.13054342567920685, 0.10214576870203018, -1.279383659362793, 1}}, follow = true}
display3D = <optimized out>
#2 0x00007fcc8f0d44a3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3 0x00007fcc8eea7fd4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#4 0x00007fcc8ef285bc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) print _M_ptr
Cannot access memory at address 0x18
(gdb) print nullptr
No symbol "nullptr" in current context.
(gdb) up 1
#1 xrt::auxiliary::tracking::slam::slam_tracker_ui::ui_runner (this=0x55625c8d2780, T_w_i_init=..., cal=..., conf=...)
at /home/linus/dev-priv/vr/basalt-deps/basalt/src/monado/slam_tracker_ui.hpp:224
224 if (img_data[cam_id].img) {
(gdb) l
219
220 if (curr_vis_data && curr_vis_data->opt_flow_res && curr_vis_data->opt_flow_res->input_images) {
221 auto &img_data = curr_vis_data->opt_flow_res->input_images->img_data;
222
223 for (size_t cam_id = 0; cam_id < num_cams; cam_id++) {
224 if (img_data[cam_id].img) {
225 img_view[cam_id]->SetImage(img_data[cam_id].img->ptr, img_data[cam_id].img->w, img_data[cam_id].img->h,
226 img_data[cam_id].img->pitch, fmt);
227 }
228 }
(gdb) print cam_id
$1 = 1
(gdb) print num_cams
$2 = 4
(gdb) print img_data[cam_id].img
value has been optimized out
(gdb) print img_data[cam_id]
value has been optimized out
(gdb) print img_data
$3 = <optimized out>
(gdb)
I'm not quite sure, not familiar enough with modern C++ specifics involved here, why it segfaults here. It somehow segfaults when trying to check img_data[cam_id].img for booleaness? (I'm also a bit confused why gdb tells me that img_data is optimized out.)
monado-service, libbasalt.so and monado-service-core.dump:
https://speicher.hamburg.freifunk.net/d/2f0f422666154eeb9798/
Edited by Linus Lüssing