slam/basalt: SIGABRT, assertion: "did not expect numerical failure during linearization"
After a few minutes of using Monado with SLAM tracking with an HP Reverb G2 I received a crash/SIGABRT with the following error message:
$ env XRT_DEBUG_GUI=on XRT_COMPOSITOR_COMPUTE=true ~/.local/bin/monado-service
...
WARN [log_frame_time_diff] Frame late by 11.76ms!
WARN [comp_renderer_draw] Compositor probably missed frame by 1.01ms
WARN [log_frame_time_diff] Frame late by 1.07ms!
WARN [log_frame_time_diff] Frame late by 11.23ms!
WARN [comp_renderer_draw] Compositor probably missed frame by 1.08ms
WARN [log_frame_time_diff] Frame late by 0.70ms!
WARN [comp_renderer_draw] Compositor probably missed frame by 1.13ms
WARN [log_frame_time_diff] Frame late by 1.03ms!
WARN [log_frame_time_diff] Frame late by 11.03ms!
WARN [comp_renderer_draw] Compositor probably missed frame by 1.09ms
WARN [log_frame_time_diff] Frame late by 1.03ms!
did not expect numerical failure during linearization
***** Assertion (numerically_valid) failed in void basalt::SqrtKeypointVioEstimator<Scalar_>::optimize() [with Scalar_ = float]:
/home/linus/dev-priv/vr/basalt-deps/basalt/src/vi_estimator/sqrt_keypoint_vio.cpp:1177:
zsh: IOT instruction (core dumped) env XRT_DEBUG_GUI=on XRT_COMPOSITOR_COMPUTE=true ~/.local/bin/monado-service
Coredump / GDB output:
$ coredumpctl dump > /tmp/monado-service-core-3.dump
Hint: You are currently not seeing messages from other users and the system.
Users in groups 'adm', 'systemd-journal' can see all messages.
Pass -q to turn off this notice.
PID: 157748 (monado-service)
UID: 1000 (linus)
GID: 1000 (linus)
Signal: 6 (ABRT)
Timestamp: Tue 2023-05-16 22:04:48 CEST (2min 51s ago)
Command Line: /home/linus/.local/bin/monado-service
Executable: /home/linus/.local/bin/monado-service
Control Group: /user.slice/user-1000.slice/session-114.scope
Unit: session-114.scope
Slice: user-1000.slice
Session: 114
Owner UID: 1000 (linus)
Boot ID: df567cbf08624b87a3f3e95bc0e314b8
Machine ID: c7caee4282714fb09cf533de05b81b41
Hostname: linus-lptp
Storage: /var/lib/systemd/coredump/core.monado-service.1000.df567cbf08624b87a3f3e95bc0e314b8.157748.1684267488000000.zst (present)
Size on Disk: 61.8M
Package: systemd/252.6-1
build-id: 190ce0b2be8342d9ff04dbc370139acc3e7f949f
Message: Process 157748 (monado-service) of user 1000 dumped core.
Module /home/linus/.local/bin/monado-service from deb systemd-252.6-1.amd64
Module /home/linus/dev-priv/vr/basalt-install/lib/libbasalt.so from deb systemd-252.6-1.amd64
Module libudev.so.1 from deb systemd-252.6-1.amd64
Module libsystemd.so.0 from deb systemd-252.6-1.amd64
Stack trace of thread 157782:
#0 0x00007f5be85aaccc n/a (libc.so.6 + 0x8accc)
#1 0x00007f5be855bef2 raise (libc.so.6 + 0x3bef2)
#2 0x00007f5be8546472 abort (libc.so.6 + 0x26472)
#3 0x00007f5be9a5f0ea n/a (/home/linus/dev-priv/vr/basalt-install/lib/libbasalt.so + 0x5f0ea)
ELF object binary architecture: AMD x86-64
More than one entry matches, ignoring rest.
$ gdb ~/.local/bin/monado-service /tmp/monado-service-core-3.dump
GNU gdb (Debian 13.1-2) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/linus/.local/bin/monado-service...
warning: Can't open file /memfd:pulseaudio (deleted) during file-backed mapping note processing
warning: Can't open file /dev/shm/monado_shm (deleted) during file-backed mapping note processing
warning: Can't open file /memfd:xshmfence (deleted) during file-backed mapping note processing
[New LWP 157782]
[New LWP 157756]
[New LWP 157749]
[New LWP 157755]
[New LWP 157748]
[New LWP 157754]
[New LWP 157753]
[New LWP 157750]
[New LWP 157751]
[New LWP 157752]
[New LWP 157763]
[New LWP 157780]
[New LWP 157757]
[New LWP 157775]
[New LWP 157824]
[New LWP 157758]
[New LWP 157793]
[New LWP 157773]
[New LWP 157762]
[New LWP 157789]
--Type <RET> for more, q to quit, c to continue without paging--c
[New LWP 157774]
[New LWP 157759]
[New LWP 157834]
[New LWP 157783]
[New LWP 157761]
[New LWP 157792]
[New LWP 157760]
[New LWP 157787]
[New LWP 157779]
[New LWP 157832]
[New LWP 157764]
[New LWP 157794]
[New LWP 157776]
[New LWP 157814]
[New LWP 157818]
[New LWP 158410]
[New LWP 158417]
[New LWP 157828]
[New LWP 157852]
[New LWP 157826]
[New LWP 157778]
[New LWP 157777]
[New LWP 157825]
[New LWP 157788]
[New LWP 158418]
[New LWP 157840]
[New LWP 157784]
[New LWP 157821]
[New LWP 157772]
[New LWP 158406]
[New LWP 157797]
[New LWP 157822]
[New LWP 157853]
[New LWP 157816]
[New LWP 158174]
[New LWP 158420]
[New LWP 157791]
[New LWP 157817]
[New LWP 157819]
[New LWP 158408]
[New LWP 157855]
[New LWP 157833]
[New LWP 157820]
[New LWP 158409]
[New LWP 158407]
[New LWP 158414]
[New LWP 157827]
[New LWP 157771]
[New LWP 158416]
[New LWP 157856]
[New LWP 158415]
[New LWP 157839]
[New LWP 157830]
[New LWP 157829]
[New LWP 158413]
[New LWP 158255]
[New LWP 157815]
[New LWP 158412]
[New LWP 157831]
[New LWP 157854]
[New LWP 158411]
[New LWP 157790]
[New LWP 157823]
[New LWP 158419]
warning: Section `.reg-xstate/157782' in core file too small.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/linus/.local/bin/monado-service'.
Program terminated with signal SIGABRT, Aborted.
warning: Section `.reg-xstate/157782' in core file too small.
#0 0x00007f5be85aaccc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f5b4bfff6c0 (LWP 157782))]
(gdb) bt
#0 0x00007f5be85aaccc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f5be855bef2 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f5be8546472 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f5be9a5f0ea in basalt::assertionFailed (expr=<optimized out>,
function=0x7f5be9f944f8 "void basalt::SqrtKeypointVioEstimator<Scalar_>::optimize() [with Scalar_ = float]",
file=0x7f5be9f944a0 "/home/linus/dev-priv/vr/basalt-deps/basalt/src/vi_estimator/sqrt_keypoint_vio.cpp", line=1177)
at /home/linus/dev-priv/vr/basalt-deps/basalt/thirdparty/basalt-headers/include/basalt/utils/assert.h:66
#4 0x00007f5be9b7ac30 in basalt::SqrtKeypointVioEstimator<float>::optimize (this=this@entry=0x5610064ff180)
at /home/linus/dev-priv/vr/basalt-deps/basalt/src/vi_estimator/sqrt_keypoint_vio.cpp:1177
#5 0x00007f5be9b856cf in basalt::SqrtKeypointVioEstimator<float>::optimize_and_marg (this=0x5610064ff180, input_images=...,
num_points_connected=..., lost_landmaks=...)
at /home/linus/dev-priv/vr/basalt-deps/basalt/src/vi_estimator/sqrt_keypoint_vio.cpp:1498
#6 0x00007f5be9b8676f in basalt::SqrtKeypointVioEstimator<float>::measure (this=0x5610064ff180, opt_flow_meas=..., meas=...)
at /home/linus/dev-priv/vr/basalt-deps/basalt/src/vi_estimator/sqrt_keypoint_vio.cpp:502
#7 0x00007f5be9b897c4 in basalt::SqrtKeypointVioEstimator<float>::initialize(Eigen::Matrix<double, 3, 1, 0, 3, 1> const&, Eigen::Matrix<double, 3, 1, 0, 3, 1> const&)::{lambda()#1}::operator()() const (__closure=0x7f5b4c000b98)
at /home/linus/dev-priv/vr/basalt-deps/basalt/src/vi_estimator/sqrt_keypoint_vio.cpp:269
#8 0x00007f5be88d44a3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007f5be85a8fd4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x00007f5be86295bc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) l
24 // Insert the on load constructor to init trace marker.
25 U_TRACE_TARGET_SETUP(U_TRACE_WHICH_SERVICE)
26
27 int
28 ipc_server_main(int argc, char *argv[]);
29
30
31 int
32 main(int argc, char *argv[])
33 {
(gdb) up 1
#1 0x00007f5be855bef2 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) l
34 #ifdef XRT_OS_WINDOWS
35 u_win_try_privilege_or_priority_from_args(U_LOGGING_INFO, argc, argv);
36 #endif
37
38 u_trace_marker_init();
39 u_metrics_init();
40
41 int ret = ipc_server_main(argc, argv);
42
43 u_metrics_close();
(gdb) up 1
#2 0x00007f5be8546472 in abort () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) l
44
45 return ret;
46 }
(gdb) up 1
#3 0x00007f5be9a5f0ea in basalt::assertionFailed (expr=<optimized out>,
function=0x7f5be9f944f8 "void basalt::SqrtKeypointVioEstimator<Scalar_>::optimize() [with Scalar_ = float]",
file=0x7f5be9f944a0 "/home/linus/dev-priv/vr/basalt-deps/basalt/src/vi_estimator/sqrt_keypoint_vio.cpp", line=1177)
at /home/linus/dev-priv/vr/basalt-deps/basalt/thirdparty/basalt-headers/include/basalt/utils/assert.h:66
66 std::abort();
(gdb) l
61 char const* file,
62 long line) {
63 std::cerr << "***** Assertion (" << expr << ") failed in " << function
64 << ":\n"
65 << file << ':' << line << ":" << std::endl;
66 std::abort();
67 }
68
69 inline BASALT_ATTRIBUTE_NORETURN void assertionFailedMsg(char const* expr,
70 char const* msg,
(gdb) up 1
#4 0x00007f5be9b7ac30 in basalt::SqrtKeypointVioEstimator<float>::optimize (this=this@entry=0x5610064ff180)
at /home/linus/dev-priv/vr/basalt-deps/basalt/src/vi_estimator/sqrt_keypoint_vio.cpp:1177
1177 BASALT_ASSERT_STREAM(
(gdb) l
1172 Timer t;
1173
1174 // linearize residuals
1175 bool numerically_valid;
1176 error_total = lqr->linearizeProblem(&numerically_valid);
1177 BASALT_ASSERT_STREAM(
1178 numerically_valid,
1179 "did not expect numerical failure during linearization");
1180 stats.add("linearizeProblem", t.reset()).format("ms");
1181
(gdb) print numerically_valid
$1 = false
(gdb) print lqr
$2 = {
_M_t = {<std::__uniq_ptr_impl<basalt::LinearizationBase<float, 6>, std::default_delete<basalt::LinearizationBase<float, 6> > >> = {
_M_t = {<std::_Tuple_impl<0, basalt::LinearizationBase<float, 6>*, std::default_delete<basalt::LinearizationBase<float, 6> > >> = {<std::_Tuple_impl<1, std::default_delete<basalt::LinearizationBase<float, 6> > >> = {<std::_Head_base<1, std::default_delete<basalt::LinearizationBase<float, 6> >, true>> = {
_M_head_impl = {<No data fields>}}, <No data fields>}, <std::_Head_base<0, basalt::LinearizationBase<float, 6>*, false>> = {_M_head_impl = 0x7f5b04012130}, <No data fields>}, <No data fields>}}, <No data fields>}}
(gdb)
monado-service, libbasalt.so and monado-service-core-3.dump:
https://speicher.hamburg.freifunk.net/d/5c335eaecafa41c1b562/
Edited by Linus Lüssing