Audio over network is delayed by half a minute, but pausing/stopping is instant
I use PipeWire on my local machine to stream to network-based speakers running the PulseAudio server. This works fine if I use PulseAudio on the local machine as well, but when I use Pipewire locally instead the audio is delayed by half a minute or so. E.g. I start a video or music file and even though I can see it playing, the audio only starts playing 30 or so seconds (estimate) later. It isn't trying to catch up either by skipping part of the audio, it literally only starts the audio that late.
The strange thing is that if I pause or stop the video, the audio instantly stops as would normally be expected. But resuming again takes half a minute for the audio to play.
This is using PipeWire 0.3.32 connected to a PulseAudio server using zeroconf on a RPi3, and I can reproduce it on 2 separate machines.