endless loop resulting OOM
Submitted by Hui Peng
Assigned to poppler-bugs
Link to original bug (#104798)
Description
Created attachment 136967 a tar.gz file containing the testcase
when using tools like pdftohtml, pdftoppm, pdftops, pdftotext on the uploaded testcases, the parser gets stuck in endless loop resulting OOM.
This is the stacktrace of pdftohtml:
#0 sysmalloc (nb=nb@entry=0x8590, av=0x7ffff7792c20 <main_arena>) at malloc.c:2768
#1 0x00007ffff7444645 in _int_malloc (av=av@entry=0x7ffff7792c20 <main_arena>, bytes=bytes@entry=0x8580) at malloc.c:4135
#2 0x00007ffff7446f3e in __GI___libc_malloc (bytes=0x8580) at malloc.c:3086
#3 0x00007ffff7828458 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff7ca55c9 in Stream::makeFilter (this=this@entry=0x555576634210, name=<optimized out>, str=str@entry=0x555576634210, params=params@entry=0x7fffffffc2e0, recursion=recursion@entry=0x3, dict=dict@entry=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Stream.cc:324
#5 0x00007ffff7ca5ccd in Stream::addFilters (this=this@entry=0x555576634210, dict=<optimized out>, recursion=recursion@entry=0x3) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Stream.cc:198
#6 0x00007ffff7c95688 in Parser::makeStream(Object&&, unsigned char*, CryptAlgorithm, int, int, int, int, bool) (this=this@entry=0x5555555ccb30, dict=dict@entry=<unknown type in /home/huip/tmp/tfuzz_eval/poppler-0.62.0/build/libpoppler.so.73, CU 0x22f494, DIE 0x2330c7>, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=cryptNone, keyLength=keyLength@entry=0x30cb, objNum=objNum@entry=0x4, objGen=0x0, recursion=0x3, strict=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Parser.cc:269
#7 0x00007ffff7c95e95 in Parser::getObj (this=this@entry=0x5555555ccb30, simpleOnly=simpleOnly@entry=0x0, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=cryptNone, keyLength=keyLength@entry=0x30cb, objNum=0x4, objGen=0x0, recursion=0x2, strict=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Parser.cc:135
#8 0x00007ffff7c95ba8 in Parser::getObj (this=this@entry=0x5555555ccb30, simpleOnly=simpleOnly@entry=0x0, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=cryptNone, keyLength=keyLength@entry=0x30cb, objNum=0x4, objGen=0x0, recursion=0x1, strict=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Parser.cc:93
#9 0x00007ffff7c95cf2 in Parser::getObj (this=this@entry=0x5555555ccb30, simpleOnly=simpleOnly@entry=0x0, fileKey=0x0, encAlgorithm=cryptNone, keyLength=0x30cb, objNum=0x4, objGen=0x0, recursion=0x0, strict=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Parser.cc:120
#10 0x00007ffff7cb1dc6 in XRef::fetch (this=0x5555555ccd30, num=<optimized out>, gen=0x0, recursion=recursion@entry=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/XRef.cc:1171
#11 0x00007ffff7c8ead6 in Object::fetch (this=this@entry=0x5555555d1838, xref=<optimized out>, recursion=recursion@entry=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Object.cc:125
#12 0x00007ffff7c290aa in Dict::lookup (this=this@entry=0x5555555d1700, key=key@entry=0x7ffff7d11493 "FontDescriptor", recursion=recursion@entry=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Dict.cc:260
#13 0x00007ffff7c532f4 in GfxFont::getFontType (xref=xref@entry=0x5555555ccd30, fontDict=fontDict@entry=0x5555555d1700, embID=embID@entry=0x7fffffffc8b8) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/GfxFont.cc:343
#14 0x00007ffff7c58f5e in GfxFont::makeFont (xref=xref@entry=0x5555555ccd30, tagA=0x5555555d1520 "F1", idA=idA@entry=..., fontDict=fontDict@entry=0x5555555d1700) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/GfxFont.cc:222
#15 0x00007ffff7c5917f in GfxFontDict::GfxFontDict (this=0x5555555d1580, xref=0x5555555ccd30, fontDictRef=0x0, fontDict=0x5555555d14c0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/GfxFont.cc:2457
#16 0x00007ffff7c3c09b in GfxResources::GfxResources (this=0x5555555cd240, xref=0x5555555ccd30, resDictA=<optimized out>, nextA=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Gfx.cc:338
#17 0x00007ffff7c47efb in Gfx::Gfx (this=0x5555555d12f0, docA=<optimized out>, outA=0x5555555cd4b0, pageNum=0x1, resDict=0x5555555ce1a0, hDPI=108, vDPI=108, box=0x7fffffffcb50, cropBox=0x0, rotate=0x0, abortCheckCbkA=0x0, abortCheckCbkDataA=0x0, xrefA=0x5555555ccd30) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Gfx.cc:541
#18 0x00007ffff7c941a6 in Page::createGfx (this=this@entry=0x5555555d1220, out=out@entry=0x5555555cd4b0, hDPI=hDPI@entry=108, vDPI=vDPI@entry=108, rotate=rotate@entry=0x0, useMediaBox=useMediaBox@entry=0x1, crop=<optimized out>, crop@entry=0x0, sliceX=sliceX@entry=0xffffffff, sliceY=0xffffffff, sliceW=0xffffffff, sliceH=0xffffffff, printing=0x0, abortCheckCbk=0x0, abortCheckCbkData=0x0, xrefA=0x5555555ccd30) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Page.cc:521
#19 0x00007ffff7c9443a in Page::displaySlice (this=0x5555555d1220, out=0x5555555cd4b0, hDPI=108, vDPI=108, rotate=0x0, useMediaBox=0x1, crop=0x0, sliceX=sliceX@entry=0xffffffff, sliceY=0xffffffff, sliceW=0xffffffff, sliceH=0xffffffff, printing=0x0, abortCheckCbk=0x0, abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Page.cc:552
#20 0x00007ffff7c94708 in Page::display (this=<optimized out>, out=<optimized out>, hDPI=<optimized out>, vDPI=<optimized out>, rotate=<optimized out>, useMediaBox=<optimized out>, crop=<optimized out>, printing=<optimized out>, abortCheckCbk=0x0, abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/Page.cc:481
#21 0x00007ffff7c98e29 in PDFDoc::displayPages (this=0x5555555cc4b0, out=0x5555555cd4b0, firstPage=<optimized out>, lastPage=0x1, hDPI=108, vDPI=108, rotate=0x0, useMediaBox=0x1, crop=0x0, printing=0x0, abortCheckCbk=0x0, abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/poppler/PDFDoc.cc:513
#22 0x000055555555ef20 in main (argc=<optimized out>, argc@entry=0x2, argv=argv@entry=0x7fffffffcf78) at /home/huip/tmp/tfuzz_eval/poppler-0.62.0/utils/pdftohtml.cc:392
#23 0x00007ffff73d91c1 in __libc_start_main (main=0x55555555e4b0 <main(int, char**)>, argc=0x2, argv=0x7fffffffcf78, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffcf68) at ../csu/libc-start.c:308
#24 0x000055555555f1aa in _start ()
Attachment 136967, "a tar.gz file containing the testcase":
testcase.tgz