We sent an AI agent to 100 SaaS websites. It found the price on 90, and how to cancel on 62.
More and more of the web is read by agents working for someone: a browser agent comparing tools, a research assistant checking a price. So I wanted to know a plain thing. If an AI agent lands on your home page, can it find what a customer would ask?
I picked 100 well-known SaaS and developer-tool websites and gave a browsing agent three questions for each:
- Price: what does the cheapest paid plan cost per month?
- Cancel: how does a customer cancel a paid subscription?
- Rate limit: what rate limit applies to the public API?
The rules: start on the home page, read and click only, no logging in, no typing, no forms, at most 12 steps, and quote the page the answer came from. Every run is recorded.
The results, with the recording of every run, are on a public page.
| Found | Not found | Blocked | |
|---|---|---|---|
| Price | 90 | 1 (+5 no fixed price) | 4 |
| Cancel | 62 | 34 | 4 |
| API rate limit | 63 | 35 | 2 |
This is a test of whether the answer can be found, not whether it exists. “Not found” means this agent didn’t reach it in 12 steps. As you will see below, sometimes the page is right there.
Prices are easy, unless the page stops the agent
Five sites have no fixed-price plan to find: Replicate, AssemblyAI and Stytch charge by usage, Lemon Squeezy by transaction, Make by credits. Of the other 95, the agent found the price on 90, usually in two steps: click “Pricing”, read.
Four stopped it with a bot check. On OpenAI, Canva and Box the agent did everything right. It found the pricing link and clicked it, and the pricing page answered with Cloudflare’s “Verify you are human”. Pipedrive’s pricing link led straight to a Cloudflare page that says “Sorry, you have been blocked”. We don’t try to get past those checks, so those count as blocked.
That is a choice those companies are free to make. It does mean that an agent asked “what does ChatGPT Plus cost?” can’t read the answer from the source.
The one real miss was Heroku. The agent opened heroku.com/pricing, read it, scrolled, and left. The Eco dyno is $5 a month, on that page, in a table that stays collapsed until you press “Show Cedar Dynos”. The agent reads what is on screen, and the price wasn’t.
Cancel is the hard one
62 of 100. And the misses are not where I expected.
GitHub has an official page called “Downgrading your account’s plan”. Vercel’s Pro plan docs have a “Downgrading to Hobby” section. Netlify has “Change your pricing plan”. The agent reached none of them, in five attempts each.
Look at where it went instead. On GitHub: pricing, then “Contact Sales”, the support request page, back to pricing, the help centre, out of steps. On Vercel: pricing, the knowledge base, the sales contact page, the knowledge base again. On Netlify: support, pricing, the knowledge base, the contact page.
The pattern is that pricing pages talk about getting in, and the way out lives in the docs, one or two levels away from anything linked on the home page. A person searches for “github cancel plan” and finds it. An agent that browses from the home page, the way this one does, doesn’t.
Sites that did well tend to link billing help close to pricing. Figma, Notion, Shopify and Linear were found in four to eight steps, quoted from their help centres and docs.
API rate limits: public, or not
63 of 100. Here “not found” mixes two things: limits that are public but deep in the docs, and limits that aren’t public at all. Mistral, for one, shows its limits in the console after you sign in. Docs on a separate site make it harder: Trello’s limits live on developer.atlassian.com, and the agent only got there on a later attempt. I have not sorted all 35 misses by hand, so I won’t say which is which.
Are the “found” answers right?
I checked 15 at random against the source page, five per task. 13 were fully right. Two were partly right: for Cursor the agent quoted a line about when to cancel, not how. For Confluence it brought back Jira’s rate limit page. None were made up.
How it works
A Chromium browser driven by Playwright, at 1280×800. Two models split the work. TypeSafe’s Jev picks what to click from the links and buttons on the page. When it thinks the answer may be on the page, a Gemini Flash-Lite model reads the page and writes the answer with a quote. It can click, scroll, go back and give up, nothing else. It respects robots.txt and says who it is in its user agent (AgentArenaBot/0.1).
Every site and task got one run. The misses got two more. Then I fixed two things in the agent, pressing “reject” on cookie banners and following a link’s address when an overlay swallows the click, and ran the remaining misses twice more. Up to five attempts. If any of them found the answer, it counts. 788 runs in all, 16 to 19 October.
At the models’ paid rates the whole thing would cost $2.62, about a third of a cent a run. I ran it on free tiers.
What this doesn’t tell you
One agent, one week, a desktop browser, run from Istanbul. Prices change with where you are: monday.com showed me €9 a seat; you may see $9. A stronger agent, or one allowed to use a search engine, would find more. A site that changes its pages tomorrow may score differently. And blocking bots on a pricing page may be a perfectly good decision for a company. This only measures what it costs in one situation.
The data, and your site
Every site, task, answer, quote, source and recording is on the results page, and as JSON and CSV. The data and the agent’s code are on GitHub. If a result about your company is wrong, there is a “Wrong? Tell us” link on every row, and I will check and correct it.
If you want to know how your own site does, leave the URL. I’ll run the same three tasks and send you the result with the recordings. For now I run them by hand, so it can take a few days.