Skip to content

Add initial hand tracking for WMR devices

Mateo de Mayo requested to merge mateosss/monado:mateosss/ht-wmr into main

This MR allows WMR headset cameras to be used for hand tracking. It is still in very early stages. To try it out you can follow this file instructions.

EDIT: For the best experience, put the headset in your head, the model works best from that perspective.

A couple of comments:

  1. When integrating the hand tracker I noticed its interface is fairly different from other trackers like t_tracker_{psmv,psvr,slam}. Essentially these trackers are glorified sinks that device drivers just stream data to. I also noticed there is a stub t_tracker_hand that has a more similar interface to what I was expecting but it is not being used. I applied small changes with this in mind like the own_xfctx option or returning a sink on ht_device_create_wmr.

  2. We are using OpenCV's camera model which is slightly different from the one on Azure-Kinect-Sensor-SDK (which is probably similar to what WMR is using). In practice, these differences shouldn't be significant as detailed in wmr_hmd_create_stereo_camera_calib's documentation.

Edited by Mateo de Mayo

Merge request reports