Invalid chunk size of malloc due to error in JPXStream::init

Hi,

During fuzzing of pdftocairo, I found the following problem.

Syntax Error (339): Dictionary key must be a name object
Syntax Error (342): Dictionary key must be a name object
Internal Error: xref num 5 not found but needed, try to reconstruct<0a>
Syntax Error (339): Dictionary key must be a name object
Syntax Error (342): Dictionary key must be a name object
==108195==WARNING: AddressSanitizer failed to allocate 0xfffffffffffd8b37 bytes
==108195==AddressSanitizer's allocator is terminating the process instead of returning 0
==108195==If you don't like this behavior set allocator_may_return_null=1
==108195==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_allocator.cc:147 "((0)) != (0)" (0x0, 0x0)
    #0 0x7f6575a86631  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0xa0631)
    #1 0x7f6575a8b5e3 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0xa55e3)
    #2 0x7f6575a03425  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x1d425)
    #3 0x7f6575a89865  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0xa3865)
    #4 0x7f6575a08b4d  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x22b4d)
    #5 0x7f6575a7e5d2 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x985d2)
    #6 0x7f657501a795 in gmalloc(unsigned long, bool) /home/mshudrak/poppler/poppler_newest/goo/gmem.h:41
    #7 0x7f657501a795 in Stream::toUnsignedChars(int*, int, int) /home/mshudrak/poppler/poppler_newest/poppler/Stream.h:143
    #8 0x7f657501a795 in JPXStream::init() /home/mshudrak/poppler/poppler_newest/poppler/JPEG2000Stream.cc:234
    #9 0x7f657501bf90 in JPXStream::getImageParams(int*, StreamColorSpaceMode*) /home/mshudrak/poppler/poppler_newest/poppler/JPEG2000Stream.cc:140
    #10 0x7f6574df39d4 in Gfx::doImage(Object*, Stream*, bool) /home/mshudrak/poppler/poppler_newest/poppler/Gfx.cc:4227
    #11 0x7f6574dfafec in Gfx::opXObject(Object*, int) /home/mshudrak/poppler/poppler_newest/poppler/Gfx.cc:4167
    #12 0x7f6574de6863 in Gfx::go(bool) /home/mshudrak/poppler/poppler_newest/poppler/Gfx.cc:753
    #13 0x7f6574de7482 in Gfx::display(Object*, bool) /home/mshudrak/poppler/poppler_newest/poppler/Gfx.cc:715
    #14 0x7f6574ea5775 in Page::displaySlice(OutputDev*, double, double, int, bool, bool, int, int, int, int, bool, bool (*)(void*), void*, bool (*)(Annot*, void*), void*, bool) /home/mshudrak/poppler/poppler_newest/poppler/Page.cc:548
    #15 0x40d9e2 in renderPage /home/mshudrak/poppler/poppler_newest/utils/pdftocairo.cc:737
    #16 0x40d9e2 in main /home/mshudrak/poppler/poppler_newest/utils/pdftocairo.cc:1257
    #17 0x7f6573eff82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #18 0x4106d8 in _start (/home/mshudrak/poppler/poppler_newest/build/utils/pdftocairo+0x4106d8)

It seems like we control size of memory to be allocated by malloc. You can find the reproducer in the attachment.

raiter_issue5465.pdf

It seems like JPXStream::init ((bufSize = oLen.getInt()) doesn’t have a check for negative values of stream length thereby making it possible to allocate large memory chunk on the heap with size controlled by an attacker in the str->toUnsignedChars. This vulnerability doesn't directly lead to any kind of code execution but can be used as a second stage of House of Force attack.

Thank you!