Why CI Failure is Actually Success

Continuous Integration serves as a vital safety net that catches errors before they reach production, significantly shortening the feedback loop. While passing CI runs are essentially overhead, failing runs are the only instances where the system provides tangible value by preventing bugs. To maintain this value, developers must eliminate flaky tests and recognize that a CI failure is actually a successful detection of a mistake.

Key Points

• CI's primary value is preventing bad outcomes by catching mistakes before deployment.
• When CI passes, it acts as overhead that adds friction without changing the deployment result.
• A failing CI run represents a successful catch, whereas a passing run is just a necessary step.
• Flaky CI is the greatest threat to the system because it makes failures unreliable and ambiguous.
• The industry should reframe CI 'failures' as positive events and distinguish them from technical flakiness.

Sentiment

The community is largely sympathetic to the core thesis that CI failures are where CI earns its keep, but pushes back meaningfully on the article's oversimplifications. The strongest disagreements target the claim that flaky tests are unfixable and the assertion that passing CI is mere overhead. The tone is constructive and knowledgeable, with several commenters adding practical depth from their own engineering experience.

In Agreement

• CI failures should be celebrated as successful catches rather than dreaded — test failures reveal mistakes in understanding, whether in the implementation or the test itself
• Flaky CI undermines the system's reliability by making failures ambiguous, and the 'just rerun it' culture kills the entire purpose of CI
• The purpose of tests in general is to fail — too many tests are written to pass rather than to catch real problems
• Good tests should be coupled to behavior and the public interface, not to implementation details, so that changing an implementation doesn't require rewriting tests

Opposed

• The article is oversimplified clickbait — CI's purpose has always been catching bugs before production, this framing adds nothing new
• Passing CI is not just overhead as the article claims — it enables other developers to build on partially-complete work and provides better record keeping than deploying without a pipeline
• The article's claim that flaky tests are 'unfixable in theory' is wrong — over 90% of flakes are reproducible and fixable through strategies like deterministic test ordering and investigating root causes like database sort issues and timezone bugs
• Swapping CI status colors (green for failure) is poor UI design that contradicts well-understood conventions, even as a rhetorical device
• A test passing on retry might indicate a real concurrency bug in production code, making blind retries potentially dangerous rather than just wasteful