NetworkManager daemon hang for several seconds in libcurl / getaddrinfo()
Tested version: NetworkManager-1.36.0-0.11.fc36.x86_64, but has been happening for >1 year using different versions of NM.
Steps to reproduce
- Connect to some wifi network
- Start a WireGuard connection
- Disconnect from wifi
- NetworkManager becomes unresponsive (nmcli and other NM interfaces hang for 10-15 seconds)
Note that my system uses systemd-resolved (250.3). I currently use DNS-over-TLS with 1.1.1.1, but the bug was also reproducible without DoT.
I attached gdb to NetworkManager and found two threads sleeping in libcurl / getaddrinfo():
0x00007ffff7651c79 in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
(gdb) bt
#0 0x00007ffff7651c79 in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
#1 0x00007ffff7656c93 in __pthread_clockjoin_ex () from target:/lib64/libc.so.6
#2 0x00007ffff7ba0033 in multi_done.lto_priv () from target:/lib64/libcurl.so.4
#3 0x00007ffff7ba07f3 in curl_multi_remove_handle () from target:/lib64/libcurl.so.4
#4 0x000055555569cdbc in cb_data_complete ()
#5 0x000055555569d799 in concheck_handle_complete ()
#6 0x00005555557939df in concheck_cb ()
#7 0x000055555569ce95 in cb_data_complete ()
#8 0x000055555569d024 in _idle_cb ()
#9 0x00007ffff78396bb in g_idle_dispatch () from target:/lib64/libglib-2.0.so.0
#10 0x00007ffff783d1bf in g_main_context_dispatch () from target:/lib64/libglib-2.0.so.0
#11 0x00007ffff78922d8 in g_main_context_iterate.constprop () from target:/lib64/libglib-2.0.so.0
#12 0x00007ffff783c8ef in g_main_loop_run () from target:/lib64/libglib-2.0.so.0
#13 0x00005555555a78c5 in main ()
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7ffff64dc500 (LWP 1035) "NetworkManager" 0x00007ffff7651c79 in __futex_abstimed_wait_common () from target:/lib64/libc.so.6
2 Thread 0x7ffff6483640 (LWP 1056) "gmain" 0x00007ffff76cdfaf in poll () from target:/lib64/libc.so.6
3 Thread 0x7ffff5c82640 (LWP 1080) "gdbus" 0x00007ffff76cdfaf in poll () from target:/lib64/libc.so.6
4 Thread 0x7fffe7fff640 (LWP 9239) "NetworkManager" 0x00007ffff76ce0ae in ppoll () from target:/lib64/libc.so.6
5 Thread 0x7fffe6ffd640 (LWP 9240) "NetworkManager" 0x00007ffff76ce0ae in ppoll () from target:/lib64/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff6483640 (LWP 1056))]
#0 0x00007ffff76cdfaf in poll () from target:/lib64/libc.so.6
(gdb) bt
#0 0x00007ffff76cdfaf in poll () from target:/lib64/libc.so.6
#1 0x00007ffff789224d in g_main_context_iterate.constprop () from target:/lib64/libglib-2.0.so.0
#2 0x00007ffff783ab40 in g_main_context_iteration () from target:/lib64/libglib-2.0.so.0
#3 0x00007ffff783ab91 in glib_worker_main () from target:/lib64/libglib-2.0.so.0
#4 0x00007ffff7867322 in g_thread_proxy () from target:/lib64/libglib-2.0.so.0
#5 0x00007ffff76550b7 in start_thread () from target:/lib64/libc.so.6
#6 0x00007ffff76da800 in clone3 () from target:/lib64/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff6483640 (LWP 1056))]
#0 0x00007ffff76cdfaf in poll () from target:/lib64/libc.so.6
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff5c82640 (LWP 1080))]
#0 0x00007ffff76cdfaf in poll () from target:/lib64/libc.so.6
(gdb) bt
#0 0x00007ffff76cdfaf in poll () from target:/lib64/libc.so.6
#1 0x00007ffff789224d in g_main_context_iterate.constprop () from target:/lib64/libglib-2.0.so.0
#2 0x00007ffff783c8ef in g_main_loop_run () from target:/lib64/libglib-2.0.so.0
#3 0x00007ffff7a9af2a in gdbus_shared_thread_func.lto_priv () from target:/lib64/libgio-2.0.so.0
#4 0x00007ffff7867322 in g_thread_proxy () from target:/lib64/libglib-2.0.so.0
#5 0x00007ffff76550b7 in start_thread () from target:/lib64/libc.so.6
#6 0x00007ffff76da800 in clone3 () from target:/lib64/libc.so.6
(gdb) thread 4
[Switching to thread 4 (Thread 0x7fffe7fff640 (LWP 9239))]
#0 0x00007ffff76ce0ae in ppoll () from target:/lib64/libc.so.6
(gdb) bt
#0 0x00007ffff76ce0ae in ppoll () from target:/lib64/libc.so.6
#1 0x00007ffff4a0c9fe in fd_wait_for_event () from target:/lib64/libnss_resolve.so.2
#2 0x00007ffff4a15172 in varlink_call.constprop () from target:/lib64/libnss_resolve.so.2
#3 0x00007ffff4a166a5 in _nss_resolve_gethostbyname4_r () from target:/lib64/libnss_resolve.so.2
#4 0x00007ffff76c265e in gaih_inet.constprop () from target:/lib64/libc.so.6
#5 0x00007ffff76c2d46 in getaddrinfo () from target:/lib64/libc.so.6
#6 0x00007ffff7b79b57 in getaddrinfo_thread.lto_priv () from target:/lib64/libcurl.so.4
#7 0x00007ffff7b742ec in curl_thread_create_thunk.lto_priv () from target:/lib64/libcurl.so.4
#8 0x00007ffff76550b7 in start_thread () from target:/lib64/libc.so.6
#9 0x00007ffff76da800 in clone3 () from target:/lib64/libc.so.6
(gdb) thread 5
[Switching to thread 5 (Thread 0x7fffe6ffd640 (LWP 9240))]
#0 0x00007ffff76ce0ae in ppoll () from target:/lib64/libc.so.6
(gdb) bt
#0 0x00007ffff76ce0ae in ppoll () from target:/lib64/libc.so.6
#1 0x00007ffff4a0c9fe in fd_wait_for_event () from target:/lib64/libnss_resolve.so.2
#2 0x00007ffff4a15172 in varlink_call.constprop () from target:/lib64/libnss_resolve.so.2
#3 0x00007ffff4a166a5 in _nss_resolve_gethostbyname4_r () from target:/lib64/libnss_resolve.so.2
#4 0x00007ffff76c265e in gaih_inet.constprop () from target:/lib64/libc.so.6
#5 0x00007ffff76c2d46 in getaddrinfo () from target:/lib64/libc.so.6
#6 0x00007ffff7b79b57 in getaddrinfo_thread.lto_priv () from target:/lib64/libcurl.so.4
#7 0x00007ffff7b742ec in curl_thread_create_thunk.lto_priv () from target:/lib64/libcurl.so.4
#8 0x00007ffff76550b7 in start_thread () from target:/lib64/libc.so.6
#9 0x00007ffff76da800 in clone3 () from target:/lib64/libc.so.6