Constant XRuns / Stutter with external USB DAC

So far I've been using pipewire successfully on my laptop with an integrated Intel DAC. I have a Numark NS6II DJ Controller which features a 4.0 Soundcard (1 Stereo pair for the main out and one pair for the pfl headphone out). Unfortunately, no matter what API is used, PW produces lots of underruns. Though I'm not sure whether those are really underruns, because it doesn't seem like the missed buffers are rejected but rather the player is given the chance to take longer than expected to fill the buffer fully. I've tried using Jack, PA, or the alsa bridge and all produce the same results. The soundsource being used also doesn't seem to make a difference. I've tried firefox, totem and Mixxx. Though when inspecting the log from Mixxx, I was able to spot the following error:

[W][000204713.517280][remote-node.c:650 client_node_port_use_buffers()] Failed to mlock memory 0x7fb6e11a5000 32832: This is not a problem but for best performance, consider increasing RLIMIT_MEMLOCK

I don't know how to increase that or even what I should increase it too. I'm running the binaries from the fedora package repositories so currently PW version 0.3.10