As we don't know how many output buffers we need to operate, we need to
avoid pool that can't grow. Otherwise the pipeline may stall, waiting
for buffers. For now, we require it to be able to grow to at least
32 buffers, which I think is a fair amount of buffers for decoders.