Optionally bail after number-of-failures threshold

It would be great to have an option to abort the run after a set number of failures. If your CI run contains 2 or 3 new failures, then it’s good to see them all so you can reason about them. If you have like 200 new fails, then something is fundamentally wrong, so we should bail on pre-merge testing until you have a more reasonable fail, rather than waiting out 10,000 crashes.