Commit d350bd2c authored by Pekka Paalanen's avatar Pekka Paalanen
Browse files

octave: add transfer_function_lut1d_error_taps.m

This script plots the maximum absolute error versus number of taps, when
you are approximating a transfer function with a 1D LUT using a given
number of taps.

This shows that increasing the number of taps has diminishing returns,
especially for pure power-law inverse functions, while the approximation
error remains relatively high even for 8-bit values.
Signed-off-by: Pekka Paalanen's avatarPekka Paalanen <>
parent 188876f5
Pipeline #494300 passed with stage
in 11 seconds
% SPDX-FileCopyrightText: 2022 Collabora, Ltd.
% SPDX-License-Identifier: MIT
% When an analytical formula is approximated with a 1D LUT, there some error
% introduced. This script studies the effect of the number of taps in a 1D
% look-up table vs. the maximum absolute error found over the curve. You
% can choose which transfer function to study, and what numbers of taps to
% test. The result is shown as a graph.
% AdobeRGB_inv_EOTF is one of the most difficult functions, being a pure
% power-law function with an infinite slope at zero. To understand why,
% see the transfer_function_lut1d_error script.
% The maximum absolute error discovered here is trying to find the maximum
% over all input values. In real applications though, input values are more or
% less quantized, which means that with enough taps the maximum error may not
% be hit. However, for inverse EOTF, the input value is an optical value
% that may be a result from another EOTF, color space conversion and/or
% blending, which makes the quantization at the inverse EOTF input
% unpredictable.
% The transfer function to investigate, as named in transfer_function().
tf_name = "AdobeRGB_inv_EOTF";
% Sequence of number of taps to iterate through.
num_taps = 17:3:1400;
% -------------------------------------------
maxerr = zeros(size(num_taps));
for i = 1:length(num_taps)
% Instantiate the tap positions.
taps = [0 : (1 / (num_taps(i) - 1)) : 1];
assert(length(taps), num_taps(i));
% Compute LUT values, the naive way
lut = transfer_function(taps, tf_name);
% The test points. 50 points per linear section to have good coverage.
x = [0 : 1/(50*(num_taps(i) - 1)) : 1];
% The reference results.
ref = transfer_function(x, tf_name);
% 1D LUT results, linearly interpolated.
intp = interp1(taps, lut, x, "linear");
% maximum absolute error
maxerr(i) = max(abs(intp - ref));
titl = sprintf("Error of approximating %s with a 1D LUT", tf_name);
plot(num_taps, maxerr);
hold on;
plot(num_taps([1 end]), [1 1] ./ 255);
xlabel("Number of taps in 1D LUT");
ylabel("Maximum absolute error");
title(titl, "Interpreter", "none");
legend("Approximation error", "1 / 255 level");
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment