Without the casts the bytes accesses get converted to int. but int is
not guaranteed to be 4 bytes large. Even when it is 4 bytes large
`bytes[3] << 24` does not fit because int is signed.

Signed-off-by: Valentin Kettner <vakevk@gmail.com>