How Footprint tests a product

What a test is, how its numbers are counted, and what the results can and can’t tell you.

What a test is

Footprint gives AI coding agents a small, real task with a developer product, for example “create a table, insert a row and read it back” with Neon. Each agent works alone on a fresh project folder, the way a developer’s coding assistant would: it searches the web, reads docs, installs packages, writes code and runs it. Every action is recorded as a step in its trail.

• Two versions. Half the agents get only the task. The other half are also told: “I don’t have an account and can’t sign up right now — please get it working anyway if there’s a legitimate way to do that without me creating an account, entering personal info, or paying.” Comparing the two shows whether agents find a no-signup option on their own.
• Rules for the agents. They may not sign up, log in, enter personal information, pay, or get around a CAPTCHA. Without a credential they use a made-up key, so the product’s real answer is still recorded.
• A fixed definition of real success per product, such as “a row read back from a real Neon database”. Only the analyst sees it, never the agents. A local stand-in or a different free tool never counts.

How the numbers are counted

An analyst model reads every trail and judges each run against the success definition. Every count shown, on this site and in the reports, is then computed by code from those per-run judgments, never written by the AI. When the analyst’s text contains a number that disagrees with the code’s count, code replaces it.

How the stories are checked

• Every step of a plain-English story must point at a real step of that run’s trail; code drops any that don’t.
• The “page to change” must be a page the agents actually read.
• A problem with a product must be backed by a direct quote from its pages or the exact error its programs printed.
• If the cause is not clear, the story says so.
• For the 11 benchmark products, the summary sentences on each results page (where the agents got stuck, why, and what to change) were checked by hand against the benchmark write-up. Where the analyst got one wrong or blamed the product for our test computer, the write-up’s wording is used instead. Each run’s own story is still written by the analyst and checked by code.

Checks done by code, not AI

For every test, code checks the product’s llms.txt, re-requests every link the agents used, looks for signup walls, checks whether each page the agents read mentions a no-signup option, and flags agents that said “it worked” while their last program failed.

What is never blamed on the product

Problems with our test computer are shown in grey as “Problem with our test computer, not the product” and left out of the numbers: network hangs, disk or port trouble, a busy AI service, Docker missing, Windows quirks, and runs that saw another run’s data. When a whole product’s result may depend on our test computer, as with Supabase, which needs Docker, its runs stay in the numbers exactly as counted and the product is labelled “Uncertain” with the reason. Links the agents guessed and home-made local fakes are never product problems either. “Account required by design” is a product choice, not a bug.

The October 2026 benchmark

• Two AI coding agents, DeepSeek V4.1 Flash and GLM 5.3 Flash, tried to set up each product on their own on 20–21 October 2026.
• Each agent tried every product twice: once with the normal instruction, and once also told “I don't have an account and can't sign up right now”. A second try was added only where a result was uncertain. 66 runs in total.
• Each product had a fixed definition of real success, for example “a row read back from a real Neon database”. Only the analyst saw it; the agents never did.
• The agents worked on a Windows computer with free web search. They could install npm packages and run Node.js, but could not sign up, log in or pay.
• Every number on this page was counted by code from the per-run results, not taken from the AI analyst's text.

Limits

• Small samples: 4 to 8 runs per product, so a single run changes a rate a lot.
• Cheaper agents: both are “Flash” models. Stronger coding agents may do better.
• Free web search often failed (DuckDuckGo blocked most searches and Bing often returned nothing useful), so agents opened docs directly. This may have hidden some no-account options.
• Our test computer runs Windows without Docker, which blocks Supabase's local option, and a tool on it broke QStash's local server in some runs.
• Inngest is not ranked: its first attempt had one mixed-up run, and the re-run was stopped at the owner’s request because it was too slow.
• One WorkOS run probably succeeded but was not counted by the analyst. The numbers here are left exactly as code counted them.

The full write-up has every piece of evidence; the benchmark page shows it product by product.