Fix gcc warnings with -Wpointer-sign

Fixes fontconfig/fontconfig#184
3 jobs for issue/184 in 8 minutes and 35 seconds (queued for 2 seconds)
Status Job ID Name Coverage
  Build
passed #894386
mingw-build

00:03:39

passed #894384
shared-build

00:05:12

passed #894385
static-build

00:08:35