Zvariant cannot deserialize a byte array and another sequence

The following code

let ctxt = Context::<LE>::new_dbus(0);
let value: (Vec<u8>, HashMap<String, Value<'_>>) = (Vec::new(), HashMap::new());
let value = zvariant::to_bytes(ctxt, &value).unwrap();
let (ba, hm): (&[u8], HashMap<&str, Value<'_>>) = zvariant::from_slice(&value, ctxt).unwrap();

panics with:

panicked at 'called `Result::unwrap()` on an `Err` value: Message("invalid type: character `y`, expected `v`, `a` or `(`")'

when the value is attempted to be de-serialized. The overall signature of the value is (aya{sv}). I was also able to recreate this with 2 byte vectors ((ayay)).

I don't understand de-serialization or parsing stuff much, so some help on this one would be greatly appreciated. From an initial look parsing the ay byte array doesn't appear to advance the signature parser enough and the hash map parser is encountering the y, but making the deserialize_ay() method advance the signature parser twice didn't seem to help.