A8gent
All articles

Business

Agent Washing: How to Check an AI Agent Vendor's Claims

Gartner estimates only about 130 of the thousands of vendors claiming agentic AI are the real thing. Here is a buyer's checklist that tests the product instead of the pitch, written for owners who are about to spend money.

A8gentAutomation practiceAugust 4, 202612 min read
Contents
  1. What agent washing is, and why the term came from an analyst firm
  2. Step one: turn the marketing claim into a testable sentence
  3. Step two: take the demo away from the demo
  4. Step three: the seven questions that separate operators from marketers
  5. Red flags, green flags, and the ones that look like the wrong colour
  6. Step four: design a pilot that can actually fail
  7. The question underneath all of it: buy, build, or neither
  8. FAQ

What agent washing is, and why the term came from an analyst firm

In June 2025 Gartner published a forecast that got quoted everywhere: more than 40% of agentic AI projects will be cancelled by the end of 2027, as organisations struggle with rising costs, unclear business value and inadequate risk controls. That is the line that made the headlines.

The line underneath it is more useful to a buyer. Gartner also identified "agent washing" - where vendors rebrand existing AI assistants, chatbots, or robotic process automation tools as "agentic AI" without delivering true agentic capabilities - and estimated that of the thousands of vendors claiming agentic solutions, only about 130 actually offer genuine agentic features.

Two things follow from that. First, the base rate is against you: if you are looking at a product because its marketing says "agentic," the prior probability that the label is doing real work is low. Second, and this is the part most buyer guides get wrong, the label is not the thing you should care about. A well-built deterministic workflow that solves your problem is worth more than a genuinely autonomous agent that solves a different one. The reason to detect agent washing is not purity. It is that a washed product is usually priced as if it were autonomous while performing as if it were a chatbot, and the difference shows up in your staffing.

So the checklist below is not "is this a real agent." It is "does this do my job, at the reliability and price claimed." Those are answerable questions, and you can answer most of them before signing anything.

Step one: turn the marketing claim into a testable sentence

Almost every AI agent pitch contains one load-bearing sentence and a lot of scaffolding. Find the sentence. It usually looks like "our agent handles X automatically" or "resolves Y% of Z."

Then rewrite it in a form that can be checked, filling in the parts the original left out:

  • Over what population? "Resolves 80% of support tickets" is meaningless without knowing which tickets. Eighty percent of all inbound, or eighty percent of the tier-one subset the vendor selected?
  • Measured how? Resolved as in the customer stopped replying, or resolved as in a human confirmed the outcome was correct?
  • Over how many attempts? A figure from one pass over a curated set is a different animal from a figure that held across repeat runs. This is the gap we walk through in what agents can and cannot do reliably, and it is the most common place a good number hides a bad system.
  • At what human cost? If 80% is handled but every one of those still gets read by staff, the saving is review time, not headcount.

Send the rewritten sentence back to the vendor and ask them to confirm or correct it. This is a genuinely diagnostic exercise. Vendors running real deployments answer with specifics and caveats, because they have watched the number move. Vendors who cannot answer will reframe the question, and that reframing is your data.

Three phrases deserve particular attention. "Up to" describes a ceiling that was reached once. "Accuracy" without a task definition is unanchored - accurate at what, judged by whom. And "fully autonomous" paired with no number is the tell that there is no number.

Step two: take the demo away from the demo

A demo is a rehearsed performance on inputs chosen for it. That is not dishonest - it is what a demo is - but it means the demo tells you almost nothing about your situation. The fix is to change the inputs.

Bring twenty to fifty of your own real cases. Pull them from your actual history, not from what you wish your inbox looked like. Include the ones your own staff find annoying: the customer who asked three questions in one message, the address with a typo, the invoice with a handwritten amendment, the person who replied to a two-year-old thread. This is the long tail, it is a large share of real volume, and it is exactly what a curated demo excludes.

Run the same case five times and compare the outputs. Not five different cases - the same case, five times. If the answers differ materially between runs, you have measured run-to-run variance yourself, in your own workflow, for free. That single test tells you more about your future support burden than any reference call will.

Give it an input it cannot possibly satisfy. A request that needs information nobody has recorded. The correct behaviour is to stop and ask, or to escalate. The failure mode is to produce something confident and plausible. You want to see which one you get before a customer does.

Break a dependency. Ask what happens when the CRM is unavailable or an API returns an error mid-run. Watch out for the phrase "handles it gracefully" and press on what it means concretely. The honest answer is usually that the run should stop and a human should be told - an agent that carries on past a failed tool call is an agent that is guessing, and it is one of the failure modes we cover in the mistakes businesses make.

Ask to see the trace. Which tools were called, with what inputs, what came back, and what the system decided. If a vendor cannot show you that for a run you just watched, they cannot show it to you when something goes wrong in production either. That is an observability problem, and it is the difference between a system you can debug and one you can only restart.

Step three: the seven questions that separate operators from marketers

These are ordered by how much they reveal per minute spent. Ask them of a person who has built the thing, not only of a salesperson.

1. "What is your autonomous completion rate on a workflow like mine, and what is the denominator?" Any vendor with production deployments has this number and will hedge it appropriately. The revealing answer is not a low number - low numbers are often fine - it is no number.

2. "What does it do when it is not confident?" You are listening for a specific mechanism: a confidence threshold, a rule, a human queue. "It asks for clarification" is a behaviour, not a mechanism, and behaviours are not reliable unless something enforces them.

3. "What write permissions does it need, and can we start read-only?" Every write action is your exposure. A vendor who resists a read-only or draft-only first phase is telling you their product has no value without full access, which is itself worth knowing. Least privilege is not paranoia here - it is the mitigation the security standards recommend for agents that read untrusted content, as we cover in defending agents against prompt injection.

4. "How would we find out it did something wrong?" Not whether it can. How you would learn. If the answer routes through a customer complaint, the monitoring layer does not exist.

5. "What happens to our data, and where does it go?" Which model provider, which region, whether your inputs are retained or used for training, and whether there is a data processing agreement. If you have customers in regulated sectors this is a procurement gate, not a preference - see agent security and privacy.

6. "If we stop paying you, what do we keep?" Prompts, workflow definitions, logs, integrations. The honest answer for most hosted products is "not much," which is fine as long as you priced the switching cost in. This is vendor lock-in and it is a cost, not a moral failing.

7. "What is the all-in monthly cost at our volume, including model usage and failed runs?" Seat pricing and usage pricing behave very differently as you scale, and failed or retried runs consume tokens without producing output. Ask for a worked example at your numbers, then compare against typical market pricing before you accept it.

Red flags, green flags, and the ones that look like the wrong colour

Red flags. A demo that cannot accept your inputs. Metrics without denominators. Resistance to a paid pilot on a real workflow. No trace or log view. An implementation timeline that assumes your data is clean. A claim to automate an entire department rather than a process. And the quiet one: a product whose only interface is chat, sold as though it takes actions - if the human is still doing every step and the model is just drafting text, you are buying a writing tool at automation prices.

Green flags. A named limitation, offered unprompted. A completion rate below 100% quoted without embarrassment. Willingness to start read-only. A visible escalation path with a named owner. Reference customers doing something structurally similar to you, not merely similar in size. And a discovery conversation that spends more time on your exceptions than on their roadmap.

The flags that fool people. Two things get read as negatives when they are usually positives. The first is a vendor saying "this part still needs a human." That is not a weak product; on current evidence it is a well-designed one, and human-in-the-loop review is the cheapest reliability mechanism available. The second is a product that turns out to be mostly a deterministic workflow with a model at one or two decision points. That is not a scandal - it is how most reliable automation is actually built, and it drifts far less between runs than a free-roaming agent. Judge it on whether it does the job, not on how autonomous it sounds. Our comparison of agents against workflow tools and hiring lays out when each is genuinely the right call.

And one that fools people the other way: an impressive live demonstration of autonomy on a task you did not choose. Watching an agent browse the web and complete a purchase is genuinely impressive and tells you nothing about whether it can reconcile your invoices.

Step four: design a pilot that can actually fail

Most pilots are structured so they cannot produce a clear no. Someone runs the tool for a month, everyone agrees it was interesting, and the decision gets made on enthusiasm. Design against that.

Pick one workflow with a countable outcome. Not "improve customer service." Something like: inbound quote requests get a first response, with the right price band, within fifteen minutes. High volume, low risk, and an outcome someone can check without a meeting. Our tool evaluation guide covers scoring candidate processes if you are choosing between several.

Write the baseline down before you start. How many of these a month, how long each takes now, how often it goes wrong now, what the current cost per item is fully loaded. Without this, any post-pilot number is unfalsifiable - and a surprising share of the value of doing it is that measuring the current process usually reveals a fix that needs no AI at all.

Set the pass mark in writing, in advance. A specific completion rate on a specific case set, with an error ceiling. Both numbers, not just the first: a system handling 90% of cases with a 15% error rate is worse than one handling 60% with a 2% error rate, because the second one you can trust and the first one you have to check anyway.

Fix the case set before the vendor sees it. Freeze your fifty cases up front. A case set that evolves during the pilot becomes a case set the product was tuned to, and you will have measured the tuning rather than the product.

Cap the spend and time. Thirty days, a fixed budget, a named decision date. If it needs six months to show signal on one workflow, that is the answer.

Agree what happens to the work. Who owns the prompts, the workflow definitions and the integration config at the end - whether you continue or not. This is a five-minute conversation before and an expensive one after. We put it in writing on every engagement, which is why our build service says plainly that you own what we build.

The question underneath all of it: buy, build, or neither

Agent washing is a detection problem, but the decision it feeds into is a comparison problem. You have three real options and the marketing only ever pitches one.

Buy a product when your workflow is genuinely standard, the vendor has customers doing exactly it, and the pricing works at your volume. You get maintenance and improvements you do not pay for individually. You accept the switching cost and the roadmap being someone else's.

Build it - yourself, or with someone - when the workflow is specific to how you operate, when it spans systems no single product covers, or when the per-seat price of the product exceeds what a fixed-scope build costs to run. On an open platform like n8n, with model costs paid at API rates, the running cost of a narrow workflow is frequently a fraction of a per-seat subscription. You own the result and you own the maintenance.

Do neither, yet when the process is not written down, the data is not clean, or nobody can say what a correct outcome looks like. This is a real answer and it is the right one more often than anyone selling either option will tell you. Automating an undefined process produces a faster undefined process. If measuring the baseline in your pilot design reveals that the actual problem is a missing field in your CRM, fix that and bank the win.

What connects all three is the same instinct the checklist runs on: compare the claim to the mechanism. What actually happens, step by step, when a case arrives? Who checks it? What happens when it breaks? A vendor who can answer those in specifics is worth talking to whatever label they use. A vendor who answers in adjectives is selling the label.

If you want an outside read on a specific pitch before you commit, that is precisely what our free automation audit does - we look at the workflow, the numbers and the vendor's claims together, and tell you plainly if the honest answer is "do neither, yet." We are a small operation, so we will also tell you where a larger firm is the better fit.

FAQ

What is agent washing?
It is Gartner's term for vendors rebranding existing AI assistants, chatbots or robotic process automation tools as 'agentic AI' without delivering genuinely agentic capability. In its June 2025 forecast that over 40% of agentic AI projects would be cancelled by the end of 2027, Gartner estimated that of the thousands of vendors claiming agentic solutions, only about 130 actually offer genuine agentic features.
Does it matter if a product is a 'real' agent or just a workflow?
Far less than the marketing implies. A deterministic workflow with a model at one or two decision points is often more reliable than a free-roaming agent and cheaper to run. What matters is whether it completes your work at the claimed rate and price. Agent washing is worth detecting mainly because washed products tend to be priced as autonomous while performing as assisted, which shows up in your staffing rather than your invoice.
What is the fastest way to test a vendor's claims?
Two tests, both free. Run twenty to fifty of your own messy real cases rather than the vendor's demo scenarios, and run the same case five times to see whether the output is stable. Curated demos hide both the long tail and the run-to-run variance, and those two things are where most of the post-purchase disappointment lives.
What should I ask about pricing before signing?
Ask for an all-in monthly figure at your actual volume, including model usage and failed or retried runs, since retries consume tokens without producing output. Then ask how the price behaves if volume doubles - seat pricing and usage pricing diverge sharply at scale. Compare the result against typical market rates rather than against the vendor's own framing.
Is it a bad sign if a vendor says a step still needs a human?
No - it is usually a good one. Human review is the cheapest reliability mechanism available for agent systems, and vendors who name their limitations unprompted are generally the ones who have run real deployments. The worrying answer is the opposite: full autonomy claimed with no completion rate attached and no escalation path described.
How long should an AI agent pilot run?
Thirty days on one workflow, with a written baseline, a frozen case set, a pass mark agreed in advance and a named decision date, is enough to get a clear answer for most business processes. If a vendor needs six months to show signal on a single workflow, that is itself the result. Cap the spend as well as the time so the pilot cannot quietly become a deployment nobody decided on.

If this resonated

Module 02 is the long version of this post.

Eval set, refusal pattern, kill switch. Build it in a weekend.

All postsLast verified 2026-08-04

Done for you

Want this built for your business?

Tell us the workflow. We scope, build, and ship the agent with guardrails and the numbers to prove it worked. The scoping call is free.

Free scoping call. You own the code.

Request a scoping call