Crash due to allocation failing...
Migrated from: [SAVANNAH-58626]
Sebastian Rasmussen reported:
A crash was found in FreeType 2.10.0 using gcc 9.3.0 ASAN and a custom allocator that controls when malloc returns NULL.
This is the gdb backtrace (relevant for Freetype) where the allocator returns NULL:
#1 (closed) 0x0000555555b1b2b8 in ft_mem_qalloc (memory=0x621000002918, size=40, p_error=0x7fffffff3c80)
at freetype/src/base/ftutil.c:76
#2 (closed) 0x0000555555b1b13d in ft_mem_alloc (memory=0x621000002918, size=40, p_error=0x7fffffff3d20)
at freetype/src/base/ftutil.c:55
#3 (closed) 0x0000555555b8a4f5 in cf2_stack_init (memory=0x621000002918, e=0x617000001210, stackSize=48)
at freetype/src/psaux/psstack.c:62
#4 (closed) 0x0000555555b83c1c in cf2_interpT2CharString (font=0x617000001208, buf=0x7fffffffaa80, callbacks=0x6170000012b8,
translation=0x7fffffffa940, doingSeac=0 '\000', curX=0, curY=0, width=0x7fffffffa930)
at freetype/src/psaux/psintrp.c:600
#5 (closed) 0x0000555555b740c8 in cf2_getGlyphOutline (font=0x617000001208, charstring=0x7fffffffaa80, transform=0x7fffffffaa40,
glyphWidth=0x7fffffffaa30) at freetype/src/psaux/psfont.c:527
#6 (closed) 0x0000555555b761a3 in cf2_decoder_parse_charstrings (decoder=0x7ffffffface0,
charstring_base=0x62e00000138b "m\213\336\367\200\334\367f\332\001\340\355\003\370k\026\336\373\264\367\200\367q\334\373r\367f\367\235\a\230\332\005\374\v\375E\006\016*\240v\367\327\331\367e\332\001\340\354\003\370P\371E\025\373\373\375E\354\367\327\367`\331\373`\367e\367\215\006\016\365\201\327\367r\332\367\222\330\001\302", , charstring_len=40)
at freetype/src/psaux/psft.c:435
#7 (closed) 0x0000555555b31467 in cff_slot_load (glyph=0x613000000c88, size=0x60b000000518, glyph_index=38, load_flags=10)
at freetype/src/cff/cffgload.c:441
#8 (closed) 0x0000555555b2de16 in cff_glyph_load (cffslot=0x613000000c88, cffsize=0x60b000000518, glyph_index=38, load_flags=10)
at freetype/src/cff/cffdrivr.c:193
#9 (closed) 0x0000555555afec90 in FT_Load_Glyph (face=0x61a000000c88, glyph_index=38, load_flags=10)
at freetype/src/base/ftobjs.c:949
This is the corresponding backtrace where the crash occurs:
#1 (closed) 0x5606e0b5f5e4 in cf2_stack_init freetype/src/psaux/psstack.c:70
#2 (closed) 0x5606e0b58c1b in cf2_interpT2CharString freetype/src/psaux/psintrp.c:600
#3 (closed) 0x5606e0b490c7 in cf2_getGlyphOutline freetype/src/psaux/psfont.c:527
#4 (closed) 0x5606e0b4b1a2 in cf2_decoder_parse_charstrings freetype/src/psaux/psft.c:435
#5 (closed) 0x5606e0b06466 in cff_slot_load freetype/src/cff/cffgload.c:441
#6 (closed) 0x5606e0b02e15 in cff_glyph_load freetype/src/cff/cffdrivr.c:193
#7 (closed) 0x5606e0ad3c8f in FT_Load_Glyph freetype/src/base/ftobjs.c:949
Looking at the source code (https://git.savannah.gnu.org/cgit/freetype/freetype2.git/tree/src/psaux/psstack.c#n62)
reveals that the logic is such that if stack->buffer fails to be allocated the code
continues to dereference the stack variable which is NULL, giving rise to the crash.
I will shortly contribute a patch fixing this.