Cutting CI/CD false positives from 45% to under 1%

Early in my career, when I was first getting familiar with the whole SDLC, I noticed something within the company.
Nobody trusted a red build.

When a pipeline failed, the default reaction across engineering was to hit retry and go get coffee. Almost half of our failures were false positives and everyone knew it, so a red build meant nothing. People retried without reading the logs. In a system where a bad deploy has real consequences, that’s a dangerous habit to have.

We fixed it in about six weeks. This post covers how: the measurement, the buy-in, the actual fixes.

Step zero: measure it

You can’t argue anyone into caring about flakiness with anecdotes. Everyone already has their own anecdote about the pipeline eating a perfectly good merge request, and after the third retelling they all blur together. And not many people really care about engineers saying “it’s annoying”.

So the first thing I did was pipe our GitLab pipeline data into Datadog and build a dashboard: failure rates over time, broken down by job and by test suite. 45% of tests failed. Even on pipelines that became green after data clean up and retries.

The dashboard made the problem undeniable, and it also made it specific. We could see exactly which suites were the worst offenders and which teams owned them. I used it to raise the issue internally, then presented it to all of engineering. With the data on a screen, buy-in was quick. We formed a small task force to work through the backlog, and I led it.

This was the first time I ever tried to convince people of my idea. I’m happy my first instinct was to back myself up with data.
If you’re planning to do something similar at your company, build the dashboard first. Every conversation afterwards gets so much easier.

What we found

The false positives fell into three buckets.

The biggest was shared environments.
Multiple test suites ran against the same environment and stepped on each other’s data. The clearest example: a test that modified a record, then used the application’s own search to verify the modification. It had worked for years. Then the environment got busier, other tests created more and more data, and eventually our record was pushed onto page two of the search results. The test looked at page one, didn’t find its record, and reported the modification as failed. Nothing was actually broken. It took someone a while to figure that out, because who suspects pagination?

The second bucket was ancient test data.
Our oldest tests reused fixture data that lived in the environment permanently. Over the years that data drifted and got mangled by other tests, and every so often a human had to go in and repair it by hand so the suite would pass again. At that point you don’t really have an automated test suite anymore.

The third was timing.
Tests that assumed an asynchronous operation had finished when sometimes it hadn’t. Individually small, but there were a lot of them.

What we did

Roughly in order of impact:

Made test data ephemeral. Every run now creates its own data and deletes it afterward. Nothing persists between runs, nothing is shared. This single change killed the pagination failure and the entire manual-repair problem. Also when the pipeline ran it did an additional sweep to clean up the environment.

Separated environments. Suites that used to collide got their own isolated environments. Together with ephemeral data, this emptied out the first bucket almost completely.

Pushed tests down the pyramid. A lot of the flakiness lived in end-to-end tests that had no reason to be end-to-end. Where the behavior under test really belonged to one service, we rewrote the e2e test as an integration or unit test. Fewer moving parts means fewer ways for the environment to interfere, and the pipelines got faster as a bonus.

Added backoff to retries. We already retried failed tests three times. It didn’t help much, because an immediate retry tends to re-enter the same race window that caused the failure. Exponential backoff made the retries actually effective against transient timing issues. I want to be careful here though: retries paper over flakiness rather than removing it, and a flaky test with retries is mostly just a slower flaky test. This is why the other three fixes came first.

Band-aid interim solutions

Some teams genuinely could not spare the time to fix their flakiest tests. Instead of pretending otherwise, we marked those specific tests as “allowed to fail”: their results stayed visible but stopped blocking pipelines, and they went into the owning team’s backlog for when capacity freed up.

I think this was the right call. A quarantined test is honest about its status. A test that fails half the time and blocks everyone’s pipeline teaches the whole organization to ignore failures, which is the disease we were treating in the first place. And because the quarantine list was explicit and visible, it kept shrinking.

Teams didn’t like seeing their names on it.

Keeping it fixed

The technical fixes got us the number, but you need ownership too. Every team got a dashboard scoped to the tests in their own domain, showing their worst offenders and their progress over time. Flakiness stopped being a vague company-wide condition and became a concrete item on each team’s board. We put alerts on top of the dashboards so regressions get caught before they become the new normal.

Six weeks after the first dashboard went up, false positives were under 1%.

The metric I actually care about is a different one. Pipelines now ran on every developer branch and on the deploy branch without anyone manually babysitting them, and when a build goes red, people stop and read it. That trust was the goal.
(The percentage is just easier to impress people with.)

If you’re starting this at your company

Measure first.

Pipe your CI data somewhere you can chart it and classify the failures before fixing anything, because the data will tell you where to aim and it will win the buy-in argument for you.

Then go after shared, persistent test data, which in our case was the single biggest source of ghosts.
Move flaky end-to-end tests down to integration or unit level where possible (this should also become the new mindset, if not e2e test will bloat again).
Use retries as a painkiller while you fix root causes, and make the failure data visible per team, because without an owner the numbers drift right back up.

None of this required exotic tooling. The gap between 45% and 1% was mostly a decision that false positives were a real problem, some data to prove it, and some elbow grease to fix it.

---

Comments