aggregator: Include min-upstream-latency in buffering time

While we can fixe the upstream latency using the min-upstream-latency, we are now forced to use queues (hence more thread) in order to store the pending data whenever we have an upstream source that has lower latency.

This fixes the issue by allowing to buffer the fixed upstream latency. This is particularly handy on single core systems were having too many threads can cause serious performance issues.