By moving the functionality down to the testclock, the implementation
no longer needs to poll the waits, but rather wait properly for
them to be added.

The performance-hit here would be that by polling the test-clock
regularly, you would create contention on the testclock-lock, making code
using the testclock (gst_clock_id_wait) fighting for the lock.