HDR10 CLL metadata is allowed to change
Contrary to popular belief HDR10 static metadata can change per segment on seamless branching Blu-rays (ripping of which was also very broken due to wrong TrueHD duplicated frames deletion on segment border) and it is also allowed to change in mp4. As ffmpeg mail says (right now last HDR metadata patches are being accepted): https://www.mail-archive.com/ffmpeg-devel@ffmpeg.org/msg108319.html
in that some formats allow use to set it on frame, some on the stream, some on the container
I notice that color data can be set per frame and per stream already and I don’t fully understand how these interact if converting between data in frame (e.g HEVC SEI in stream in hev1) or data in header (e.g. MOV mdcv tag or HEVC SEI in hvc1 format).
And this is correct. Free CTA-861-G (and now -H) standard mandate that and I quote:
If the Source supports the transmission of the Dynamic Range and Mastering InfoFrame and if it determines that the Sink is capable of receiving that information, the Source shall send the Dynamic Range and Mastering InfoFrame once per Video Field
What that means is that it is allowed to change and display should be fast enough to change it on the fly.
In fact there is a sample from "In the Heart of the Sea" movie https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/uploads/16c628c535865d7282a48317064345a2/out.mp4 that utilizes that.
It can cause all kind of issues for you. mpv does support it and so does MadVR.