Otherwise what can happen is the following: Consider two live sources, and one of the live sources can only answer the latency query once it has data to forward. This might take a few seconds, while the other source is already ready to produce data.
Aggregator would now do a latency query once the early source produced data, fail the query and then wait forever until the late source is also producing data. Only then it would do a latency query again, which would now succeed. Aggregator would have waited longer than the latency deadline of the early source and would have dropped a lot of data unnecessarily while waiting for the late source, but instead it could've already output data for the early source.