aggregator: Include min-upstream-latency in buffering time

While we can fixe the upstream latency using the min-upstream-latency, we
are now forced to use queues (hence more thread) in order to store the pending
data whenever we have an upstream source that has lower latency.

This fixes the issue by allowing to buffer the fixed upstream latency. This is
particularly handy on single core systems were having too many threads can
cause serious performance issues.

Part-of: <!684>
34 jobs for 1.18 in 2 minutes and 16 seconds (queued for 6 seconds)