Skip to content

Draft: drm/msm: sparse / "VM_BIND" support

Rob Clark requested to merge robclark/msm:sparse into msm-next

Conversion to DRM GPU VA Manager, and adding support for Vulkan Sparse Memory in the form of:

  1. A new VM_BIND submitqueue type for executing VM MSM_SUBMIT_BO_OP_MAP/MAP_NULL/UNMAP commands
  2. Extending the SUBMIT ioctl to allow submitting batches of one or more MAP/MAP_NULL/UNMAP commands to a VM_BIND submitqueue

The UABI takes a slightly different approach from what other drivers have done, and what would make sense if starting from a clean sheet, ie separate VM_BIND and EXEC ioctls. But since we have to maintain support for the existing SUBMIT ioctl, and because the fence, syncobj, and BO pinning is largely the same between legacy "BO-table" style SUBMIT ioctls, and new-style VM updates submitted to a VM_BIND submitqueue, I chose to go the route of extending the existing SUBMIT ioctl rather than adding a new ioctl.

I also did not implement support for synchronous VM_BIND commands. Since userspace could just immediately wait for the SUBMIT to complete, I don't think we need this extra complexity in the kernel.

The corresponding mesa MR: mesa/mesa!32533

Notes/TODOs/Open Questions:

  1. The first handful of patches are from Bibek Kumar Patro's series, iommu/arm-smmu: introduction of ACTLR implementation for Qualcomm SoCs, which introduces PRR (Partially-Resident-Region) support, needed to implement MAP_NULL (for Vulkan Sparse Residency)
  2. Why do VM_BIND commands need fence fd support, instead of just syncobjs? Mainly for the benefit of virtgpu drm native context guest<->host fence passing, where the host VMM is operating in terms of fence fd's (syncobs are just a convenience wrapper above a dma_fence, and don't exist below the guest kernel).
  3. Currently shrinker support is disabled (hence this being in Draft/RFC state). To properly support the shrinker, we need to pre-allocate various objects and pages needed for the pagetables themselves, to move memory allocations out of the fence signaling path. This short-cut was taken to unblock userspace implementation of sparse buffer/image support.
  4. Could/should we do all the vm/vma updates synchronously and defer only the io-pgtable updates to the VM_BIND scheduler queue? This would simplify the previous point, in that we'd only have to pre-allocate pages for the io-pgtable updates.
  5. Currently we lose support for BO dumping for devcoredump. Ideally we'd plumb MSM_SUBMIT_BO_DUMP flag in a MAP commands thru to the resulting drm_gpuvas. To do this, I think we need to extend drm_gpuva with a flags field.. the flags can be driver defined, but drm_gpuvm needs to know not to merge drm_gpuvas with different flags.

This MR can be found in patch form if you prefer: https://patchwork.freedesktop.org/series/142263/

Edited by Rob Clark

Merge request reports

Loading