There are ffmpeg native decoder that should be drained if they work on multi-threads (thread cnt must be greater than 1.), even if AV_CODEC_CAP_DELAY is not set on codec_ctx.
In my case, when using the ffmpeg native PNG decoder(avdec_png) or VP9 decoder(avdec_vp9), the frame is not completely drained. There was delay capabilities is not set in these codecs. Used libavcodec version is 58.35.100.