Audio stops for a while when there is an automatic change in profile

I have a setup with speakers connected to digital output of my sound card via S/PDIF. When I'm switching to headphones plugging in to the front jack port and vice-versa, when unplugging headphones, the playback just stops for around 10s (I think the "reconnect" time varies a bit between trials). Note that the playback isn't muted - it just stops and doesn't proceed untill the audio is back. The only way to speed-up this "reconnect" is to change volume level a bit (I currently am just muting and unmuting it).

The profile auto-switching between digital output for speakers and analog for headphones didn't work at all before 0.4.6. This functionality was merged in !276 (merged) (I was actually waiting for this to be released).

My setup:

Host: Arch Linux
Kernel: Linux Arch 5.15.13-zen1-1-zen
Pipewire: 0.3.43
Wireplumber: 0.4.6

My audio card (lspci -v -s <card>):

00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio
	DeviceName: Onboard - Sound
	Subsystem: ASRock Incorporation Device 1220
	Flags: bus master, fast devsel, latency 32, IRQ 167
	Memory at df540000 (64-bit, non-prefetchable) [size=16K]
	Memory at df520000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [50] Power Management version 3
	Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

I've also captured the logs from wireplumber via WIREPLUMBER_DEBUG=4 wireplumber >wp_log 2>&1 when switching from headphones to speakers (unplugging happens at 23:16:40): wp_log

pw-dump without headphones plugged in: pw_dump