Footprint

Benchmark Download as Markdown

About this document. This is the benchmark’s original write-up from 21 October 2026, published unchanged. “The owner” is Niranjan, who ran the benchmark. File names such as reports/… and CHANGELOG.md refer to the Footprint project, and “Stage 1” is the set of fixes made on 22 October 2026. Section 6 notes where those fixes changed a verdict in section 5; Clerk’s problem, for example, is now kept. Web addresses are shown as text: they record what the agents read on 20 and 21 October, and some pages have moved since. Footprint’s website shows the same numbers product by product.

Footprint benchmark, October 2026

Can AI coding agents set up 11 developer products on their own, without an account?

1. What was done

2. Headline findings

3. Leaderboard

Rank Product Category Reached real success (A) Reached real success (B) Found no-account option without hint (A) With hint (B) False success claims Environment problems Biggest problem (one line)
1 Upstash QStash † Background jobs 2 of 4 4 of 4 4 of 4 4 of 4 0 of 8 0 Uncertain: the SDK's dev mode could not unpack its local server on our machine ("tar: Cannot connect to D: resolve failed")
2 Neon Databases 0 of 2 2 of 2 0 of 2 2 of 2 0 of 4 0 Setup pages read without the hint don't mention Claimable Neon, so agents used made-up credentials
3 Prisma Postgres † Databases 2 of 4 0 of 4 2 of 4 1 of 4 3 of 8 0 Most agents used the local npx prisma dev server, which this test's definition does not count
4 Auth0 Authentication 0 of 4 0 of 4 0 of 4 0 of 4 4 of 8 0 Needs an Auth0 account: both setup routes need a login (CLI or Dashboard); no account-free path
4 Clerk Authentication 0 of 4 0 of 2 1 of 4 1 of 2 4 of 6 0 Account-free setup is advertised, but the CLI refuses it for Express
4 Postmark Email 0 of 2 0 of 2 0 of 2 0 of 2 1 of 4 0 Needs a server token from the Postmark dashboard; every run got HTTP 401
4 Resend Email 0 of 2 0 of 2 0 of 2 0 of 2 0 of 4 0 Needs an API key made in an account; every run got 401 "API key is invalid"
4 SendGrid Email 0 of 2 0 of 2 0 of 2 0 of 2 1 of 4 0 Needs a SendGrid account and API key; every run got HTTP 401
4 Supabase † Databases 0 of 3 0 of 4 0 of 3 2 of 4 3 of 7 0 Its account-free path needs Docker, which our Windows test machine does not have
4 Trigger.dev Background jobs 0 of 2 0 of 3 0 of 2 0 of 3 1 of 5 0 Needs a login ("You must login first"); every run got "Invalid API key"
4 WorkOS † Authentication 0 of 4 0 of 4 2 of 4 0 of 4 3 of 8 0 The Node SDK page sends agents to the dashboard and does not point to the account-free option

4. Category comparisons

Databases

Authentication

Email

Background jobs

5. Product details

Upstash QStash (Background jobs), rank 1 †

Neon (Databases), rank 2

Prisma Postgres (Databases), rank 3 †

Supabase (Databases), rank 4 (tie) †

Clerk (Authentication), rank 4 (tie)

WorkOS (Authentication), rank 4 (tie) †

Auth0 (Authentication), rank 4 (tie)

Resend (Email), rank 4 (tie)

Postmark (Email), rank 4 (tie)

SendGrid (Email), rank 4 (tie)

Trigger.dev (Background jobs), rank 4 (tie)

6. Messages worth sending

Recommended by Footprint, with strong evidence:

Not recommended by Footprint, but the evidence looks strong (worth a human look before sending):

Recommended by Footprint, but not listed:

No messages were sent. The files in reports/ are Footprint's raw output; read section 7 before using them.

7. Problems with the test itself

8. Cost and time

Product Result used Tokens Minutes Extra attempts (tokens, minutes)
Upstash QStash re-run, analysed again 2,049,980 16.6 attempt 1: 852,109, 23.0
Neon attempt 1 353,906 3.9 none
Prisma Postgres attempt 1 2,114,316 31.1 cut-off re-run: 205,264, 14.8; re-run: 435,349, 56.8
Auth0 attempt 1 1,081,604 12.2 none
Clerk attempt 1 1,270,303 17.0 none
Postmark attempt 1 611,182 4.5 none
Resend attempt 1 270,076 7.9 none
SendGrid attempt 1 185,727 1.5 none
Supabase attempt 1 861,272 19.2 none
Trigger.dev attempt 1 733,886 12.8 none
WorkOS attempt 1 2,047,127 22.0 none
Inngest (skipped) not ranked none none attempt 1: 2,006,043, 11.0; cancelled re-run: 690,733, 9.4
Total 11,579,379 148.7 4,189,498 tokens, 115.0 minutes