CarLookout was a buyer-side used-car search engine. It scraped 56 dealer sites across Georgia and Alabama, normalized every listing to a common schema, and ran each vehicle's photos through a local vision model to write descriptions, pull features off the window sticker, and flag condition notes the dealer had not bothered to type in.
The design constraint was deliberate and slightly unusual: build it so a machine can read it
as easily as a person can. Every vehicle got a human HTML page and a matching JSON endpoint
at the same slug. Clean URLs, structured data, no login, no rate limit, no obfuscation, a real
sitemap, and an honest robots.txt.
Then we told nobody. No launch post, no social, no ads, no submissions, no backlinks. The question was simple: if you build something genuinely legible to Agents and simply leave it sitting there, what finds it?
Something found it. Just not the something that pays for a website.
Part oneAgents go straight for the agent surface
Every vehicle existed at two addresses. /vehicles/2023-bmw-x3-… returned a rendered page
for a person. /api/vehicles/2023-bmw-x3-… returned the same car as JSON. Nothing pointed
at the API. It was not in the sitemap, not linked in the navigation, and not mentioned in
robots.txt. It was discoverable only by noticing the pattern.
Over a 14-day window with full origin logs, one crawler noticed.
| ClaudeBot, 14 days | Count |
|---|---|
| Total requests | 249,671 |
| HTML vehicle pages read | 16,308 |
| JSON API objects read | 16,308 |
| Outbound dealer links followed | 16,220 |
| Distinct images pulled | 182,701 |
Those two highlighted numbers are not a typo and not a rounding. For every single one of the 16,308 vehicles it read as a page, ClaudeBot also pulled the same vehicle as structured JSON. A perfect 1:1. It did not scrape the HTML and move on. It found the machine-readable surface and preferred it, then followed the outbound link to the dealer as well.
That is 65% of the entire catalog, ingested both ways, by one crawler, in two weeks, from a site with no inbound links.
The interesting part is what happened when we ran the same measurement across every other crawler that showed up.
| Crawler | HTML | JSON API | API share |
|---|---|---|---|
| ClaudeBot | 16,496 | 16,326 | 49% |
| GoogleOther | 925 | 694 | 42% |
| Amazonbot | 37 | 27 | 42% |
| GPTBot | 153 | 72 | 32% |
| OAI-SearchBot | 49 | 8 | 14% |
| Googlebot | 841 | 12 | 1% |
| Bingbot | 125 | 2 | 1% |
| Applebot | 9,650 | 0 | 0% |
Requests to human pages vs. the undocumented JSON API, same 14-day window, same origin logs.
API share is API requests as a percentage of that crawler's total vehicle requests.
There is a clean split in that table, and it does not fall along the lines you would expect from a company's size or sophistication.
The crawlers that feed Agents and answer engines went looking for structured data. ClaudeBot, GoogleOther (which feeds Gemini and AI Overviews rather than the blue links), Amazonbot, and GPTBot all found an endpoint nothing advertised, and all routed a large fraction of their requests through it.
The crawlers that feed classic search indexes did not. Googlebot made 841 vehicle requests and hit the API twelve times. Bingbot, twice. Applebot crawled 9,650 pages, more than any other non-Anthropic crawler on the site, and touched the JSON endpoint exactly zero times.
Googlebot and GoogleOther are run by the same company against the same site in the same fortnight. One used the API for 42% of its requests. The other used it for 1%. The difference is not capability. It is what the crawler is for.
This is the finding worth carrying forward. "Machine readable" has quietly split into two different things. A search crawler wants a document it can rank and excerpt. An Agent crawler wants an object it can parse and act on. Optimizing for one does not get you the other, and a site can be a total non-entity in Google while being comprehensively ingested by the systems people increasingly ask before they ever open a search box.
If you are building anything you want Agents to consume, the practical lesson is that the surface gets found on pattern alone. We never published the API. It was inferred from the URL structure and then crawled exhaustively. Consistency in your routing is discoverability.
Part twoThe bill came, the buyers did not
So the experiment worked. The site was found, read, understood, and traversed by exactly the class of software it was designed for. Now the uncomfortable half.
| 30 days of traffic | Figure |
|---|---|
| Total requests | 372,048 |
| Pageviews | 79,338 |
| Classified non-human | 97.1% |
| Egress | 20 GB |
| Egress, worst single day | 16.3 GB |
| Human vehicle views (8 days) | 24 |
On 22 July alone, one crawler pulled 211,116 requests and 16.3 GB in a single day, which is more bandwidth than the site's entire human audience consumed over its whole life by several orders of magnitude.
Of the 2.9% that looked like a browser, most still was not a person. Roughly three quarters of it was
a single synthetic user-agent hitting only /, /search, and
/alerts in a fixed pattern. Nearly all of the remainder was WordPress exploit probing:
requests for /wp-admin, /config/env.php, and
/.git/logs/refs/heads/main on a site that has never run WordPress and never used Git in
its document root.
Strip all of that out and the genuine human audience was 24 vehicle-detail views across eight days.
And here is the part that actually settles the question. We logged referrers on every human session.
| Referrer | Events |
|---|---|
| Direct / none | 2,659 |
| carlookout.com (internal) | 84 |
| google.com | 10 |
| ChatGPT, Claude, Perplexity | 0 |
Referrer for all non-bot events, 28 June to 28 July 2026.
Zero. Not a low number. Zero.
Something on the order of a quarter of a million requests went into reading this catalog on behalf of AI systems, and across a full month not one identifiable human arrived from any of those AI systems. Whatever the Agents learned about 16,308 cars in Georgia and Alabama, none of it came back as a person looking at a car.
What it cost
The ingest side was not free either. Enrichment ran a vision model over every listing: 24,065 of
25,087 vehicles processed, split between gemma4:26b and gemma4:12b on a
single RTX 3090. That card sat at 356 W and 75 °C essentially continuously for a month, roughly
250 kWh, before counting what the air conditioning spent pulling that heat back out of a closet in
Alabama in July. The image pipeline produced 307,668 WebP files totalling 30 GB on disk, and it was
that image corpus, 182,701 distinct files of which were pulled by a single crawler, that drove most
of the bandwidth.
None of those are catastrophic numbers on their own. A few tens of dollars of electricity, some bandwidth, a disk. The point is the ratio. Every one of those dollars bought machine comprehension, and machine comprehension converted to nothing.
The conclusion we did not want
The optimistic reading of this experiment, and the one we went in expecting, was that being maximally legible to Agents is an early-mover distribution strategy. Build clean structured data, let the Agents find it, and eventually they send you people.
The data does not support the second half. It strongly supports the first. Agents will find you, without marketing, without backlinks, without an announcement, purely on the shape of your URLs. They will read everything. In the current arrangement they will not send anyone back.
That may change. Answer engines are still deciding how and whether to attribute and link out, and the economics of that decision are being argued about in public right now. But as of July 2026, on real traffic, from a real site, the honest summary is this: being perfectly readable to Agents is a cost centre, not a traffic strategy. Build for it because you think the arrangement will change, or because you want the option. Do not build for it expecting visitors this year.
CarLookout answered its question in 33 days for about the price of a tank of gas. That is a cheap experiment and a genuinely useful answer, which is why the ingest is off, the GPU is doing something else, and this page is all that is left.
Method and caveats
- Crawler identity was verified, not assumed. User-agent strings are trivially
spoofed. Of 249,671 requests claiming to be ClaudeBot, 249,667 originated from
216.73.216.0/22, which appears on Anthropic's published crawler list atclaude.com/crawling/bots.json. Three other addresses claiming ClaudeBot sent one request each and are excluded. - It was crawling, not thrashing. The 249,671 requests covered 231,537 distinct URLs with zero query strings, so this was not a crawler stuck in a faceted-search trap re-reading the same pages.
- Two different windows. Origin access logs retain 14 days, so the per-crawler HTML-vs-API breakdown covers 14 July to 28 July. The traffic, egress, and referrer totals come from Cloudflare and the application's own event log, which cover the full 30 days.
- Human numbers are an upper bound. The 2,659 direct events include our own testing and monitoring, which we did not exclude. The real human figure is lower than stated, not higher. The 24 vehicle-detail views is the conservative Cloudflare-side count.
- One site, one vertical, one month. This is a single observation, not a controlled study. The crawler split in part one was consistent and large enough to be worth reporting. The zero-referral result in part two is specific to this site's total obscurity, and a site with existing authority would likely see different behaviour.