And only ever use the non-live values if all pads are non-live,
otherwise only use the results of all live pads.

It's unclear what one would use the values for in the non-live case, but
by this we at least pass them through correctly then.

This is a follow-up for 794944f7, which
causes wrong latency calculations if the first pad is non-live but a
later pad is actually live. In that case the live values would be
accumulated together with the values of the non-live first pad,
generally causing wrong min/max latencies to be calculated.