Skip to content
  • wm4's avatar
    avutil/pixdesc: deprecate AV_PIX_FMT_FLAG_PSEUDOPAL · d6fc031c
    wm4 authored
    PSEUDOPAL pixel formats are not paletted, but carried a palette with the
    intention of allowing code to treat unpaletted formats as paletted. The
    palette simply mapped the byte values to the resulting RGB values,
    making it some sort of LUT for RGB conversion.
    
    It was used for 1 byte formats only: RGB4_BYTE, BGR4_BYTE, RGB8, BGR8,
    GRAY8. The first 4 are awfully obscure, used only by some ancient bitmap
    formats. The last one, GRAY8, is more common, but its treatment is
    grossly incorrect. It considers full range GRAY8 only, so GRAY8 coming
    from typical Y video planes was not mapped to the correct RGB values.
    This cannot be fixed, because AVFrame.color_range can be freely changed
    at runtime, and there is nothing to ensure the pseudo palette is
    updated.
    
    Also, nothing actually used the PSEUDOPAL palette data, except xwdenc
    (trivially changed in the previous commit). All other code had to treat
    it as a special case, just to ignore or to propagate palette data.
    
    In conclusion, this was just a very strange old mechnaism that has no
    real justification to exist anymore (although it may have been nice and
    useful in the past). Now it's an artifact that makes the API harder to
    use: API users who allocate their own pixel data have to be aware that
    they need to allocate the palette, or FFmpeg will crash on them in
    _some_ situations. On top of this, there was no API to allocate the
    pseuo palette outside of av_frame_get_buffer().
    
    This patch not only deprecates AV_PIX_FMT_FLAG_PSEUDOPAL, but also makes
    the pseudo palette optional. Nothing accesses it anymore, though if it's
    set, it's propagated. It's still allocated and initialized for
    compatibility with API users that rely on this feature. But new API
    users do not need to allocate it. This was an explicit goal of this
    patch.
    
    Most changes replace AV_PIX_FMT_FLAG_PSEUDOPAL with FF_PSEUDOPAL. I
    first tried #ifdefing all code, but it was a mess. The FF_PSEUDOPAL
    macro reduces the mess, and still allows defining FF_API_PSEUDOPAL to 0.
    
    Passes FATE with FF_API_PSEUDOPAL enabled and disabled. In addition,
    FATE passes with FF_API_PSEUDOPAL set to 1, but with allocation
    functions manually changed to not allocating a palette.
    d6fc031c