The atomic operations were used in an attempt to limit the amount of
wakeups done on the eventfd. It's however racy and causes trouble in
some cases.

Remove the atomic operations all together, it's probably not worth
optimizing this too much.