Frequent Linux CI failures with: Cannot initialize inotify: Too many open files
To reproduce
Submit a bunch of merge requests
Expected result
In general, tests should pass
Actual result
For example in !437 (merged):
ERROR: test-sd-activation
=========================
dbus-daemon[31711]: Cannot initialize inotify: Too many open files
...
Bail out! ERROR:../../test/test-utils-glib.c:419:test_connect_to_bus: assertion failed (error == NULL): GDBus.Error:org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. (g-dbus-error-quark, 4)
Additional context
This is inotify_init1()
or inotify_init()
failing with EMFILE
"Too many open files", documented in inotify_init(2)
as meaning one of these, probably the first:
EMFILE The user limit on the total number of inotify instances has been reached.
EMFILE The per‐process limit on the number of open file descriptors has been reached.
I think this is probably not really a dbus bug. I think this is happening because the freedesktop.org CI runners are shared between lots of jobs, and one (or more!) of the other concurrently-scheduled jobs is eating all the inotify instances - the default limit is only 1024, so it's relatively easy for that to happen.