flagset: Fail parsing on overflowing hex strings

This adds code to detect when the hex form of the string we are to parse exceeds the number of bytes that would form a 32bit flag. This will avoid treating as flagset anything above then the expected 32bit and also stop treading DRM format with modifiers as flagset (like drm-format=AB24:0x0100000000000002).

cc @slomo @thaytan @vjaquez @He_Junyan