dbus_connection_send should check to install write watch if pending messages remain
Submitted by Patrick Allison
Assigned to D-Bus Maintainers
Created attachment 28990 Process log with DBUS_VERBOSE set showing missed sent message
dbus_connection_send currently only wakes the main loop if there are remaining messages (connection->n_outgoing > 0). However, there are two ways for messages to still be remaining: 1) if the socket was not immediately writable, or 2) if the IO path was not acquirable. If 1), the transport installs the write watch, but if 2), the write watch is not installed, and while the main loop wakes up, it won't attempt to handle the write watch, and the messages won't get sent.
This can happen if the dispatch loop and the read/write loop are in different threads: the read watch handler is called, which processes data, and queues the message - this wakes the mainloop (because _dbus_connection_queue_message_link wakes the mainloop) which dispatches the message. The dispatched message gets a reply, which sends dbus_connection_send, all before the read handler ever finishes (and thus before the IO path is released).
The message is then queued for sending, but no write watch is installed, and so the message will just sit there, possibly for very long.
I'm attaching the output of a process where this happened with DBUS_VERBOSE (and PTHREAD_IN_VERBOSE) set. The log begins when the process calls dbus_watch_handle on the read watch. You can see the main thread being woken, the dispatch handling, and the send occurring in the other thread before dbus_watch_handle releases the IO path. The main loop is woken up since there are still messages to send, but has nothing to do: the write watch is disabled, and the dispatch status is complete.
Note that this process was using the dbus-cxx D-Bus bindings, which uses two separate threads for dispatching and watching read/write fds. It also was using dbus-1.2.16, but I believe the problem is still present in the 1.3 branch.
Attachment 28990, "Process log with DBUS_VERBOSE set showing missed sent message":