When a Security Check Can Never Turn Red

I found a security check last month that had passed every single time it ran, for months. Green, always green. Then I looked closer: it could not have failed if it tried. The tool running it had a fallback built in that turned any error into a passing result.

That check had one job: warn us if something looked wrong. It had quietly stopped doing that job a long time ago, and nobody noticed, because the color on the dashboard never changed.

This is not rare. Once you start looking for it, you find the same pattern everywhere teams build automated checks into a development pipeline. A rule gets written with good intentions, gets wired into a build or a deploy step, and somewhere between the intention and the execution it quietly loses its ability to say no.

I want to walk through the shapes this takes, because they repeat across very different systems, and then give you a way to test for it in your own pipeline.

A check that can't fail isn't a check

The point of a gate in a pipeline is to make a decision: does this change go through, or does it get blocked. If the answer is always yes, the gate isn't making a decision anymore. It's decoration.

The dangerous version of this isn't the missing gate. Everyone notices a missing gate eventually, usually the hard way, and then someone builds one. The dangerous version is the gate that exists, has a nice green checkmark next to it, gets referenced in onboarding docs as "we do check for this", and has never once actually stopped anything.

The shapes this takes

A few patterns I keep running into, described generically because the specific tools don't matter, the shape does.

The pipe swallows the real exit code. A script pipes its output into a compression or logging tool, and the pipeline's exit status becomes the exit status of that second tool, not the one that actually did the work. A shell pipeline reports the exit code of the last command by default, not the one that failed.

# looks safe, isn't
some_backup_tool | gzip > backup.gz
echo $?   # this is gzip's exit code, not some_backup_tool's

# actually checks the right thing
set -o pipefail
some_backup_tool | gzip > backup.gz
echo $?   # now reflects a failure anywhere in the pipe

• The scanner never looks where the problem actually lives. A secret-scanning rule gets added to catch credentials in code, but the file-type filter excludes exactly the places people paste things in a hurry: log output, generated fixtures, notebook exports. The scanner runs clean because it never opens the file with the leak.
• The test suite exercises the wrong identity. An end-to-end suite is supposed to prove that authenticated flows work correctly. If the session setup silently fails, every "authenticated" test quietly runs as an anonymous user instead, hits a login page, and still reports a pass because nothing about that outcome throws an error.
• The monitor confirms the job ran, not that it did anything. A scheduled check reports healthy because the process exited without an exception, even though the actual work it was supposed to evaluate never happened, because an earlier step in the same job aborted first.
• The rule lives only in a document. A convention gets written down: every change of this kind needs a specific label, review, or test. It works, right up until someone rewrites the onboarding doc that mentions it, and the sentence quietly disappears. Nothing enforced it, so nothing noticed it was gone.

How I test if a gate is real

None of this is about carelessness. Every one of these checks was built by someone trying to do the right thing. The problem is that a check's happy path gets exercised constantly, by definition, every time it runs, while its failure path might get tested once, when it was written, if at all.

So that's where I look now. Before I trust a gate, I try to make it fail on purpose.

• Feed it a case you know is bad. Not one you assume is bad. If a check is supposed to catch a leaked credential, put a fake one exactly where a real one would land, and confirm the run actually goes red.
• Read the config for anything that swallows failure. continue-on-error, || true, a bare except: pass, a catch block that only logs. Every one of these is a decision a human made once, usually to unblock something urgent, and it tends to outlive the reason it was added.
• Ask what "no findings" actually means. A clean scan and a scan that silently didn't run look identical on a dashboard. A gate should fail loud if it couldn't do its job at all, not report success by default.
• Check who verified the failure path, not the success path. If nobody can point to a run where this check went red on a real problem, you don't know it works. You know it has never had a reason to disagree with you yet.

The takeaway

A gate that has never once said no isn't evidence that everything is fine. It's evidence that nobody has tested whether it can say no at all.

If you run automated checks in your pipeline, whether that's security scanning, backup verification, or a simple linting rule, pick one this week and try to break it on purpose. If you can't get it to turn red, you don't have a check. You have a formality that looks like one.