fast-path: Fix some sketchy pointer arithmetic

We want a uint8_t * at the end of this math, because that's what the function we're about to pass it to takes. But ->bits is a uint32_t, so if we just do the math in units of that we can avoid the explicit factor of four which would risk an integer overflow.

Fixes: #14 (closed)