This currently results in sending data buffers before a segment event, which isn't valid in GStreamer.
It can be reproduced with a sender like:
gst-launch-1.0 videotestsrc ! matroskamux streamable=1 ! tcpserversink host=127.0.0.1 port=8888
And a receiver:
gst-launch-1.0 tcpclientsrc port=8888 ! matroskademux ! fakesink
Note that I'm totally uncertain if this is the right way to fix this. I also othough of pushing an early segment event, but it seems to break later streaming.