qtmux: Aggregator port breaks prefill mode

gst-launch-1.0 videotestsrc num-buffers=1000 ! avenc_prores ! qtmux name=mux reserved-prefill=true reserved-max-duration=60000000000 !filesink location=meh.mov    audiotestsrc num-buffers=1000 ! mux.audio_%u

Without audio it works.

CC @meh