Skip to content
  • Mathieu Bridon's avatar
    Add a tool to compare results to a reference · 0c77720c
    Mathieu Bridon authored
    This prints a "compatibility" result, for a certain definition of
    "compatibility":
    
    *   we look at the results in order, and consider we failed at the
        first result which doesn't match;
    *   for every matching result we get points;
    *   the first page is worth more point than the second, which is worth
        more points than the third, which is worth more points than the
        fourth, which is worth more points than the fifth; all pages after
        that are worth the same amount of points as the fifth;
    
    Hopefully this is not a terrible definition of "compatibility", as it
    favours the first pages of results, which really are what most people
    care about. Being wrong in the later pages is still counted as
    incompatibility, but it's not as bad as being wrong at the start.
    
    The compatibility target (threshold) can be passed as an argument, so
    that we can eventually run this in a CI and try to improve our
    compatibility.
    0c77720c