gallium/swr: Re-enable scratch space for client-memory buffers

Commit 7d33203b fixed race condition
in freeing scratch memory mechanism but that approach creates
performance regression in some cases. This change revert previous
changes and fix freeing scratch memory mechanism.

Reviewed-by: Bruce Cherniak <>
23 jobs for !4406 with swr-fix-mt-race-condition in 12 minutes and 48 seconds (queued for 3 seconds)
merge request