Yes, if your automated tests are the only thing checking your product before it ships. Automation verifies the things someone already thought to check. It cannot notice the things nobody thought of, and that is where most customer-facing bugs live.
This is not an argument against automation. Automation is better than humans at exactly one thing, and it is the thing humans are worst at.
What automation is genuinely better at
Repetition without fatigue. A suite runs 400 checks on every commit, at 3am, in the same order, forever. A person doing that would start missing things by check 40.
Catching regressions fast. If your login broke twenty minutes ago, an automated test tells you now. A person tells you next week.
Scale. Running the same checks across 30 browser and device combinations at once is not something a person can do.
If you do not have automated tests, get them. This post is not telling you otherwise.
What automation structurally cannot do
An automated test is an assertion someone wrote down. It checks that a thing you predicted still behaves the way you predicted. This makes it excellent at protecting known behaviour and blind to everything else.
Specifically, it cannot:
Notice that something looks wrong. A test asserts the confirmation message is present. It passes. The message says "Thank you, undefined." The element existed, so the test was satisfied.
Judge whether something makes sense. A three-step signup can be technically perfect and still confuse everyone who meets it. No assertion covers "a reasonable person would not understand this."
Explore. A test follows its script exactly. It never wonders what happens if you press the back button after paying, or open the same order in two tabs, or paste 500 characters into a name field.
See layout failure. Unless you have visual regression testing set up, and most teams do not, a button rendered white-on-white passes every functional check.
Test what was shipped this week. New features arrive before their tests do. The gap between shipping and covering is exactly where bugs escape.
The evidence
Stripe reviewed the checkouts of 1,600 leading e-commerce sites. 95% made five or more basic errors. More than a third of European checkouts accepted an invalid card number. 45% of APAC sites let customers try to pay with an expired card.
These are large companies. They have engineers. They have test suites. The errors survived anyway, because nobody wrote an assertion for "reject a card number that cannot exist" — it did not occur to anyone that the field would accept it.
Apple's own numbers tell the same story from a different angle. In 2024 Apple reviewed 7.77 million submissions and rejected 1.93 million. The largest single rejection category was Performance, with 1,235,471 rejections — apps that crashed, hung, or broke during review. Every one of those was submitted by a team who believed it was finished.
The split, in one table
| Automated tests | A person testing | |
|---|---|---|
| Known behaviour stays working | Excellent | Slow and expensive |
| Runs on every commit | Yes | No |
| 30 browser combinations | Yes | No |
| Notices "undefined" in a real message | No | Yes |
| Judges whether a flow makes sense | No | Yes |
| Finds bugs nobody predicted | No | Yes |
| Tests features shipped this week | Not yet | Yes |
| Spots a broken layout | Only with visual testing | Yes |
The two do not overlap. That is why the answer is not "which one" but "how much of each."
How to tell if you have the gap
Three questions.
Where did your last three customer-reported bugs come from? If your suite was green when they shipped, your suite is not covering what your customers actually do.
When did you last change a test because it kept failing on something that was fine? Suites decay. A team that has quietly disabled its flakiest tests has less coverage than it thinks.
Who last used your product as a confused person rather than as its builder? If the answer is a real customer, they are doing your exploratory testing, and they are not being paid for it.
What this looks like in practice
Keep automation for regression. Add a human pass before releases that matter — a launch, a checkout change, a payment provider switch, a redesign.
For most small teams that is a few days a month, not a full-time hire. Ours starts at $349 for a single flow, and you can see the kind of report it produces here.