Check that the coordinate transforms are at least within integer range before…

Check that the coordinate transforms are at least within integer range before using them to draw an image mask. oss-fuzz/18418
4 jobs from fix-of8418 in 39 minutes and 58 seconds (queued for 3 seconds)
Status Job ID Name Coverage
  Build
passed #32820
build

00:09:43

passed #32821
build_clang_libcpp

00:11:55

passed #32822
build_gcc_49

00:08:25

passed #32823
build_mingw64_fedora

00:09:54