loop: add back the destroy loop
We need to keep the sources around until all sources are dispatched. If not: source A and B are active after poll, A is dispatched first and removes B, when B is then dispatched, the memory is gone. We don't free the source but simply mark the fd invalid and move it do a destroy list. After all sources are dispatched we destroy. This is safe because removing a source is either done from the poll context (with invoke) or when holding the right locks.