si-audio-endpoint: don't build DSP format in _set_ports_format

Allows any format to be set, which is needed for non DSP modes.
6 jobs for ports-format-improvements in 1 minute and 18 seconds (queued for 6 seconds)
Status Name Job ID Coverage
  Container
passed container_fedora #13826764

00:00:16

passed container_ubuntu #13826765

00:00:16

 
  Build
passed build_on_fedora_no_docs #13826767

00:01:00

passed build_on_fedora_with_docs #13826766

00:01:01

passed build_on_ubuntu_no_gir #13826769

00:00:59

passed build_on_ubuntu_with_gir #13826768

00:01:00