Benchmark Download as Markdown
About this document. This is the benchmark’s original write-up from 21 October 2026, published unchanged. “The owner” is Niranjan, who ran the benchmark. File names such as reports/… and CHANGELOG.md refer to the Footprint project, and “Stage 1” is the set of fixes made on 22 October 2026. Section 6 notes where those fixes changed a verdict in section 5; Clerk’s problem, for example, is now kept. Web addresses are shown as text: they record what the agents read on 20 and 21 October, and some pages have moved since. Footprint’s website shows the same numbers product by product.
Footprint benchmark, October 2026
Can AI coding agents set up 11 developer products on their own, without an account?
1. What was done
- Footprint sent two AI coding agents, DeepSeek V4.1 Flash and GLM 5.3 Flash, to set up developer products on their own, on 20–21 October 2026.
- 12 products in 4 categories (databases, authentication, email, background jobs) were planned. 11 are reported. Inngest was skipped at the owner's request (see section 7).
- Every product used the same settings: both agents; version A (the normal instruction) and version B (the same, plus "I don't have an account and can't sign up right now…"); one run per agent and version, with a second run only where the result was uncertain. 66 runs in total.
- Each product had a fixed "real success" definition (for example "a row read back from a real Neon database"). Only the analyst model saw it; the agents never did.
- Agents worked on a Windows computer with free web search. They could install npm packages and run Node.js, but could not sign up, log in or pay.
- The analyst that writes each report was DeepSeek V4.1 Flash (GLM 5.3 Flash for Resend, after a rate limit).
- Total for the 11 reported results: 148.7 minutes of test time and 11,579,379 tokens. Including re-runs and the skipped product: 263.7 minutes and 15,768,877 tokens.
- Every count in this file is computed by Footprint's code from the per-run results, not taken from the AI analyst's text.
2. Headline findings
- Agents only reached real success where an official no-account path exists. 3 of 11 products had any successful run: Upstash QStash (6 of 8 runs, with its local dev server), Neon (2 of 4, with Claimable Neon) and Prisma Postgres (2 of 8, with
npx create-db). The other 8 had 0 counted successes: Auth0, Postmark, Resend, SendGrid and Trigger.dev need an account for any credential, Clerk's account-free mode is refused for Express, Supabase's account-free path needs Docker (missing on our machine), and one WorkOS run probably succeeded but was not counted (section 5). Overall, 10 of 66 runs reached real success. - The one-line hint barely changed how often agents found a no-account option overall: 9 of 33 runs without it, 10 of 33 with it. It helped a lot for Neon (0 of 2 without, 2 of 2 with, both finishing the task, matching the earlier Neon tests: 1 of 8 and 4 of 4). But not for WorkOS, where agents found the option in 2 of 4 runs without the hint and 0 of 4 with it. Real success was 4 of 33 runs without the hint and 6 of 33 with it.
- The setup pages agents read first often skip the no-account option. Neon's Node.js page only says "You can obtain a connection string for your database by clicking the Connect button in the Console nav." (https://neon.com/docs/serverless/serverless-driver). WorkOS's Node SDK page only says "To use the library you must provide an API key, located in the WorkOS dashboard" (https://workos.com/docs/sdks/node). Both Neon runs without the hint, and 4 of the 5 WorkOS runs that read that page, never found the no-account option, used made-up credentials and failed.
- Clerk advertises an account-free start that its CLI refuses for Express. Its docs say "run
clerk initwithout an account to get started immediately with temporary development keys" (https://clerk.com/docs/cli.md), but the CLI answered "--accountless is not supported for Express." - Agents often claim success they did not reach. In 20 of 66 runs the agent said the task worked, but the run did not reach real success (made-up keys, a home-made local fake, or a failed last run). Worst: Clerk 4 of 6, Auth0 4 of 8.
3. Leaderboard
| Rank | Product | Category | Reached real success (A) | Reached real success (B) | Found no-account option without hint (A) | With hint (B) | False success claims | Environment problems | Biggest problem (one line) |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Upstash QStash † | Background jobs | 2 of 4 | 4 of 4 | 4 of 4 | 4 of 4 | 0 of 8 | 0 | Uncertain: the SDK's dev mode could not unpack its local server on our machine ("tar: Cannot connect to D: resolve failed") |
| 2 | Neon | Databases | 0 of 2 | 2 of 2 | 0 of 2 | 2 of 2 | 0 of 4 | 0 | Setup pages read without the hint don't mention Claimable Neon, so agents used made-up credentials |
| 3 | Prisma Postgres † | Databases | 2 of 4 | 0 of 4 | 2 of 4 | 1 of 4 | 3 of 8 | 0 | Most agents used the local npx prisma dev server, which this test's definition does not count |
| 4 | Auth0 | Authentication | 0 of 4 | 0 of 4 | 0 of 4 | 0 of 4 | 4 of 8 | 0 | Needs an Auth0 account: both setup routes need a login (CLI or Dashboard); no account-free path |
| 4 | Clerk | Authentication | 0 of 4 | 0 of 2 | 1 of 4 | 1 of 2 | 4 of 6 | 0 | Account-free setup is advertised, but the CLI refuses it for Express |
| 4 | Postmark | 0 of 2 | 0 of 2 | 0 of 2 | 0 of 2 | 1 of 4 | 0 | Needs a server token from the Postmark dashboard; every run got HTTP 401 | |
| 4 | Resend | 0 of 2 | 0 of 2 | 0 of 2 | 0 of 2 | 0 of 4 | 0 | Needs an API key made in an account; every run got 401 "API key is invalid" | |
| 4 | SendGrid | 0 of 2 | 0 of 2 | 0 of 2 | 0 of 2 | 1 of 4 | 0 | Needs a SendGrid account and API key; every run got HTTP 401 | |
| 4 | Supabase † | Databases | 0 of 3 | 0 of 4 | 0 of 3 | 2 of 4 | 3 of 7 | 0 | Its account-free path needs Docker, which our Windows test machine does not have |
| 4 | Trigger.dev | Background jobs | 0 of 2 | 0 of 3 | 0 of 2 | 0 of 3 | 1 of 5 | 0 | Needs a login ("You must login first"); every run got "Invalid API key" |
| 4 | WorkOS † | Authentication | 0 of 4 | 0 of 4 | 2 of 4 | 0 of 4 | 3 of 8 | 0 | The Node SDK page sends agents to the dashboard and does not point to the account-free option |
- Ranked by real success rate across all runs, then by real success without the hint (version A). Equal numbers share a rank; the 8 products with no real success tie at 4 and are listed alphabetically.
- † = uncertain. QStash: all 4 version A runs hit a
tarerror that probably depends on our machine. Prisma Postgres: 2 runs were contaminated, and its re-run was lost to a network outage. Supabase: our machine has no Docker. WorkOS: one run probably succeeded but was not counted, so A is probably 1 of 4. Details in sections 5 and 7. - "Environment problems" counts runs that crashed on the test machine (Footprint's own count). It is 0 for every result used here; the machine-dependent problems marked † are not in this count.
4. Category comparisons
Databases
- Easiest: Neon, once agents had the hint. Both hinted agents found Claimable Neon in Neon's docs index (https://neon.com/docs/llms.txt): "If the user cannot create an account right now, follow Claimable Neon to provision a Postgres project without signup." Both created a real temporary database and read a row back.
- Without the hint, both Neon agents followed the normal setup pages and stopped at "NeonDbError: password authentication failed for user 'neondb_owner'".
- Prisma Postgres shows its no-account command in its docs: "Run
npx create-db@latest. It creates a temporary Prisma Postgres database without an account and prints a connection string plus a claim URL." (https://www.prisma.io/docs/postgres.md). The 2 runs that used it succeeded, both without the hint. - But 5 of 8 Prisma runs chose the local
npx prisma devserver instead, which this test's definition does not count as a real Prisma Postgres database. If local runs counted, Prisma might score higher (uncertain). - Supabase had no success. Its account-free route is the local stack: "That stack runs in Docker containers, so you need a container runtime installed first." (https://supabase.com/docs/guides/local-development/cli/getting-started). Its Docker-free runtime is not supported on Windows (https://supabase.com/docs/guides/local-development/docker-and-native-runtimes). Our test machine is Windows without Docker, so this result is uncertain.
Authentication
- No product reached real success by the code count. WorkOS came closest: 2 of 4 runs without the hint found its account-free option through WorkOS's llms.txt (https://workos.com/auth.md: "temporary, anonymous WorkOS environments you can provision without an account"), and one got real credentials with one
curlcall. - Clerk says "run
clerk initwithout an account to get started immediately with temporary development keys" (https://clerk.com/docs/cli.md), but for Express its CLI answered "--accountless is not supported for Express. Runclerk auth loginand useclerk init --app <app_id>instead." - Auth0 has no account-free path in the evidence. Its Express quickstart offers two setup routes, and both need a login: the CLI route checks that "you're authenticated (and prompt for login if needed)", and the other says "Go to the Auth0 Dashboard" (https://auth0.com/docs/quickstart/webapp/express).
- This category had the most false success claims: Clerk 4 of 6, Auth0 4 of 8, WorkOS 3 of 8. Agents often showed a 401 from an app with made-up keys and called it done.
- All three need an account; no run succeeded. Every run stopped at HTTP 401. Resend: "API key is invalid". Postmark: "Request does not contain a valid Server token." SendGrid: "The provided authorization grant is invalid, expired, or revoked".
- Resend was the clearest. Its quickstart tells agents "Human must first create an API key and verify their domain" (https://resend.com/docs/send-with-nodejs). It was the only email product with 0 false success claims (Postmark 1 of 4, SendGrid 1 of 4).
- Postmark's agent guide says "Obtain the token from the Postmark dashboard: Servers -> select/create server -> API Tokens" (https://postmarkapp.com/developer/get-started). SendGrid's quickstart starts with "1. Sign up for a SendGrid account." (https://www.twilio.com/docs/sendgrid/for-developers/sending-email/quickstart-nodejs).
- Uncertain: web search mostly failed during these three tests (section 7), so agents may have missed a no-account option if one exists.
Background jobs
- Easiest: Upstash QStash. Every run found its official local dev server (https://upstash.com/docs/qstash/howto/local-development:
npx @upstash/qstash-cli dev). 6 of 8 runs published a message that a local worker received, logging "hello". - QStash's Getting Started page does not mention the local server: "Prerequisite You need an Upstash account before publishing messages, create one here." (https://upstash.com/docs/qstash/overall/getstarted). Agents found the local-development page anyway.
- Trigger.dev needs a login. Its agent guide says "tell the user they will get a link to sign in or sign up and authorize the CLI" (https://trigger.dev/SKILL.md), and its CLI answered "Error: You must login first." 0 of 5 runs succeeded.
- Inngest was skipped (section 7).
5. Product details
Upstash QStash (Background jobs), rank 1 †
- Verdict: "Every agent found the no-signup option, but only 6 of 8 runs reached the goal."
- What agents did: read the local-development page → started QStash's local dev server → published a message → a local worker logged "hello". All 4 version B runs used the command-line server (
npx @upstash/qstash-cli dev) and all 4 succeeded. - Top problem (uncertain): all 4 version A runs used the SDK's dev mode, which failed to unpack its server: "tar: Cannot connect to D: resolve failed" (DeepSeek, version A,
node publish.js). That message comes from GNU tar, which was first on our machine's PATH (from Git for Windows); Windows' owntarmay not fail. 2 of these 4 runs worked around it and succeeded. - Does well: an account-free local server (https://upstash.com/docs/qstash/howto/local-development). Clear errors, e.g. "[QStash Dev] Port already in use on port 41001. Set QSTASH_DEV_PORT to use a different port, or stop the process holding it."
- Message recommended: YES by Footprint (the
tarproblem). Not listed in section 6, because it depends on our test machine. - Previous results: none.
Neon (Databases), rank 2
- Verdict: "Without a hint, agents found the no-signup option in 0 of 2 runs. With a hint, 2 of 2."
- What agents did: without the hint: search → Neon's Node.js pages → install
@neondatabase/serverless→ made-up connection string → failure. With the hint: search → https://neon.com/docs/llms.txt →npx neon claim create(Claimable Neon) → script → row read back. - Top problem: the pages read without the hint only offer the account route and never mention Claimable Neon (checked by code and by hand): "You can obtain a connection string for your database by clicking the Connect button in the Console nav." (https://neon.com/docs/serverless/serverless-driver). Result: "NeonDbError: password authentication failed for user 'neondb_owner'".
- Does well: https://neon.com/docs/llms.txt: "If the user cannot create an account right now, follow Claimable Neon to provision a Postgres project without signup. A human claims it later." Both hinted runs read a row back from a real temporary database.
- Message recommended: NO. Footprint's analyst listed no problem for a message. We think the missing pointer is worth a human look (section 6).
- Previous results: agrees. Earlier: 1 of 8 without the hint, 4 of 4 with it, all finishing. Now: 0 of 2 and 2 of 2, both finishing.
Prisma Postgres (Databases), rank 3 †
- Verdict: "Without a hint, agents found the no-signup option in 2 of 4 runs. With a hint, 1 of 4."
- What agents did: read Prisma's docs → either ran
npx create-db@latestfor a temporary hosted database (2 runs, both succeeded, both without the hint) or started the localnpx prisma devserver (5 runs) → set up Prisma ORM 7 → wrote and read a row. - Top problem (uncertain): most runs used the official local
prisma devserver instead of a hosted database. The test's definition ("a real Prisma Postgres database (official temporary databases count)") does not count local runs, which lowered the score. The hosted no-account path is documented: "Runnpx create-db@latest. It creates a temporary Prisma Postgres database without an account and prints a connection string plus a claim URL." (https://www.prisma.io/docs/postgres.md). - Does well: setup errors name the exact fix, and agents applied it: "PrismaClient was instantiated without any options. A driver adapter is required to connect to your database." and "The datasource property
urlis no longer supported in schema files. Move connection URLs for Migrate toprisma.config.ts…". - Message recommended: YES by Footprint ("quickstart does not show the Prisma ORM 7 adapter setup"). Not listed: the errors above already explain the fix, and the message text contains unfilled placeholders (section 7).
- Previous results: partly disagrees. Earlier: 6 of 6 runs without the hint found
npx create-db. Now: 2 of 4 (the others choseprisma dev). In the re-run's 2 usable runs, both found it. - Reliability: 2 of 8 runs were contaminated (a shared local port), and the re-run was lost to a network outage (section 7).
Supabase (Databases), rank 4 (tie) †
- Verdict: "Without a hint, agents found the no-signup option in 0 of 3 runs. With a hint, 2 of 4."
- What agents did: without the hint: made-up keys → "401 Unauthorized" or a project address that does not exist. With the hint: found the local Supabase stack, but
npx supabase startfailed with "failed to connect to the docker API at npipe:////./pipe/docker_engine". 3 runs then built home-made local fakes and claimed success (not counted). - Top problem (uncertain, depends on our machine): the account-free path needs Docker: "That stack runs in Docker containers, so you need a container runtime installed first." (https://supabase.com/docs/guides/local-development/cli/getting-started). The Docker-free runtime is for "coding agent sandboxes and CI runners", but its table says Windows is "Not supported. Use Docker." (https://supabase.com/docs/guides/local-development/docker-and-native-runtimes).
- Does well: the Docker requirement is stated plainly (quote above), and the API's auth error is clear: "JWT could not be decoded".
- Message recommended: YES by Footprint (no Docker-free option on Windows). Not listed: it depends on our test machine, and the message calls a hinted run "without the no-account hint".
- Previous results: none.
Clerk (Authentication), rank 4 (tie)
- Verdict: "Without a hint, agents found the no-signup option in 1 of 4 runs. With a hint, 1 of 2."
- What agents did: installed
@clerk/express, addedclerkMiddleware()andgetAuth(), used made-up keys, showed the protected route answering 401, and 4 of 6 claimed success (not real Clerk keys). One run tried Clerk's account-free CLI mode. - Top problem: account-free setup is advertised but refused for Express. https://clerk.com/docs/cli.md says "run
clerk initwithout an account to get started immediately with temporary development keys", and Clerk's own Express error page says "'npx clerk@latest init' does not need a Clerk account and writes temporary dev keys." But the CLI answered "--accountless is not supported for Express. Runclerk auth loginand useclerk init --app <app_id>instead." and "Express requires API keys — set them up manually: clerk init --app <app_id>". - Also: the Express quickstart only says "In the Clerk Dashboard, navigate to the API keys page." (https://clerk.com/docs/quickstarts/express).
- Does well: unauthenticated requests got "HTTP/1.1 401 Unauthorized" with "x-clerk-auth-status: signed-out", and the CLI's refusal names the exact alternative.
- Message recommended: NO. Footprint's quote check removed the problem ("the quote does not support the problem"). We think the evidence is strong (section 6).
- Previous results: none.
WorkOS (Authentication), rank 4 (tie) †
- Verdict: "Without a hint, agents found the no-signup option in 2 of 4 runs. With a hint, 0 of 4."
- What agents did: 2 runs without the hint read WorkOS's llms.txt, found https://workos.com/auth.md and got real temporary credentials (
curl -s -X POST https://api.workos.com/x/one-shot-environments). The others used made-up client IDs ("invalid_client / Invalid client id.") or built local fakes. - Uncertain count: one run (DeepSeek, without the hint) probably met the success definition but was marked unsuccessful. With the real temporary credentials, its protected route answered 401 {"error":"unauthorized","reason":"no_session_cookie_provided"}. That reason came from WorkOS's SDK (
authenticateWithSessionCookie), not from the agent's own code. Real success without the hint is probably 1 of 4. - Top problem: the Node SDK page sends agents to the dashboard and never mentions the account-free option: "To use the library you must provide an API key, located in the WorkOS dashboard, as an environment variable
WORKOS_API_KEY" (https://workos.com/docs/sdks/node). - Does well: https://workos.com/auth.md: "temporary, anonymous WorkOS environments you can provision without an account". It works over plain HTTP with "No authentication".
- Message recommended: YES (section 6).
- Previous results: none.
Auth0 (Authentication), rank 4 (tie)
- Verdict: "No agent reached the goal in 8 runs, with or without a hint."
- What agents did: read the Express or Node.js API quickstart, installed Auth0's SDKs and used a made-up tenant (e.g. https://fake-tenant.auth0.com). They got errors such as "You must provide a 'tokenSigningAlg' for validating symmetric algorithms" or an HTTP 500 for the missing tenant. One built its own local token server. 4 of 8 claimed success.
- Top problem: it needs an account. Both setup routes on the Express quickstart need a login: the CLI route checks that "you're authenticated (and prompt for login if needed)", and the Dashboard route says "Go to the Auth0 Dashboard" (https://auth0.com/docs/quickstart/webapp/express). No account-free option appeared in the evidence. This is by design, not a docs error.
- Does well:
express-oauth2-jwt-bearerrejected requests with no token (HTTP 401 "Authentication required"), and a precise issuer error: "discovered metadata issuer does not match the expected issuer". - Message recommended: NO (no product problem found).
- Previous results: none.
Resend (Email), rank 4 (tie)
- Verdict: "No agent reached the goal in 4 runs, with or without a hint."
- What agents did: read https://resend.com/docs/send-with-nodejs, installed
resendand ran a script with a made-up key → 401 "API key is invalid". Hinted runs looked for a sandbox and found none. All 4 reported the failure honestly. - Top problem: it needs an API key made in an account, by design: "Human must first create an API key and verify their domain" (send-with-nodejs) and "You must create at least one API key to use the Resend platform through code (e.g., SDKs, API, command-line interface (CLI), or AI tools)." (https://resend.com/docs/create-an-api-key).
- Does well: the quickstart states the human step plainly; every script reached Resend's live API; clear error "API key is invalid".
- Message recommended: YES by Footprint (it asks for a sandbox key). Not listed as a bug: it is a design choice that Postmark and SendGrid share (section 6).
- Previous results: not in the comparison list (a 20 October test also ended at 401 in every run).
Postmark (Email), rank 4 (tie)
- Verdict: "No agent reached the goal in 4 runs, with or without a hint."
- What agents did: installed
postmarkor used curl with a made-up token → HTTP 401 {"ErrorCode":10,"Message":"Request does not contain a valid Server token."}. 1 of 4 claimed success although its last run failed. - Top problem: it needs a server token from an account, by design: "Obtain the token from the Postmark dashboard: Servers -> select/create server -> API Tokens" (https://postmarkapp.com/developer/get-started).
- Does well: a "Get started with your AI agent" build spec on the same page, and a clear 401 error.
- Message recommended: NO.
- Previous results: none.
SendGrid (Email), rank 4 (tie)
- Verdict: "No agent reached the goal in 4 runs, with or without a hint."
- What agents did: read the Node.js quickstart, installed
@sendgrid/mailand used a made-up key → 401 "The provided authorization grant is invalid, expired, or revoked". 1 of 4 claimed success althoughnpm run sendfailed. This was the fastest test (1.5 minutes). - Top problem: it needs an account, by design: "1. Sign up for a SendGrid account. 2. Enable Two-factor authentication. 3. Create and store a SendGrid API Key" (https://www.twilio.com/docs/sendgrid/for-developers/sending-email/quickstart-nodejs).
- Does well: prerequisites stated up front, and the SDK warns 'API key does not start with "SG.".'
- Message recommended: NO.
- Previous results: none.
Trigger.dev (Background jobs), rank 4 (tie)
- Verdict: "No agent reached the goal in 5 runs, with or without a hint."
- What agents did: found the docs through https://trigger.dev/llms.txt, wrote a task that logs "hello", and tried to trigger it with a made-up key → "TriggerApiError: Invalid API key" (401).
npx trigger.dev devanswered "Error: You must login first." - Top problem: it needs a login, by design: "Otherwise, tell the user they will get a link to sign in or sign up and authorize the CLI." (https://trigger.dev/SKILL.md).
- Does well: agent-focused docs (llms.txt, SKILL.md), and the CLI error names the fix: "Please set the TRIGGER_ACCESS_TOKEN environment variable with a Personal Access Token."
- Message recommended: NO.
- Previous results: none.
6. Messages worth sending
Recommended by Footprint, with strong evidence:
- WorkOS: "The Node SDK setup page tells agents to get an API key from the dashboard and never mentions WorkOS's account-free one-shot environments, so runs that followed it used made-up client IDs and got 'invalid_client'." Evidence: https://workos.com/docs/sdks/node and https://workos.com/auth.md (section 5). Note: 2 of 4 runs without the hint still found the option through llms.txt.
Not recommended by Footprint, but the evidence looks strong (worth a human look before sending):
- Clerk: "Clerk's CLI docs and its Express error page say
clerk initworks without an account, but the CLI refuses account-free mode for Express." Footprint's quote check dropped this; the evidence is in section 5. Update 22 Oct 2026 (Stage 1): with the improved quote rule, Footprint now keeps this problem, andreports/clerk-message.mdrecommends it. - Neon: "The Node.js setup pages do not mention Claimable Neon, so agents without the hint used made-up credentials." This was seen in earlier tests too (1 of 8 found it without the hint).
Recommended by Footprint, but not listed:
- Prisma Postgres: the errors already explain the fix, and the message had unfilled placeholders (filled by code since Stage 1, 22 Oct 2026; its opening is now flagged for a human edit, because it says "mostly worked" while code counted 2 of 8).
- Supabase: depends on our machine having no Docker, and the message mislabels a hinted run.
- Upstash QStash: the
tarerror is probably tied to our machine's PATH. - Resend: having no sandbox key is a design choice that Postmark and SendGrid share. Footprint flagged only Resend, likely because a different analyst model wrote its report.
No messages were sent. The files in reports/ are Footprint's raw output; read section 7 before using them.
7. Problems with the test itself
- Prisma Postgres: in attempt 1, two runs used the same local port (51216, from
prisma dev's fixed ports) at the same time and were marked cross-run contamination. With the owner's approval we re-ran once, one run at a time. The first re-run was cut off when the session was interrupted (205,264 tokens). The second lost 6 of 8 runs to an internet outage (the AI provider could not be reached: "fetch failed"; one npm install timed out). As the rules say, we recorded it and kept attempt 1 (8 complete runs), marked uncertain. - Upstash QStash: attempt 1 lost its first 4 runs to a 10-minute network hang (a web search and model calls never returned). Footprint counted them as ordinary "time limit" failures. We re-ran once. The re-run's report then failed (the backup analyst's answer was cut off 3 times), so with the owner's approval we used Footprint's "Analyse again" with the usual DeepSeek analyst. The leaderboard uses this re-run.
- Inngest: attempt 1 (2,006,043 tokens) had one contaminated run: an agent started Inngest's dev server on its default port 8288 instead of its assigned ports and saw another run's function. The approved one-at-a-time re-run was stopped after 9.4 minutes at the owner's request (too slow), so Inngest is not ranked. For reference only: in attempt 1, 4 of 8 runs reached the goal (all 4 DeepSeek runs, one of them contaminated), and the quickstart still says "You need no Inngest account for this exercise." This is in line with the earlier Inngest test.
- Test machine: Windows without Docker (this blocks Supabase's local stack), with GNU tar from Git for Windows first on PATH (QStash's SDK dev mode failed with "tar: Cannot connect to D: resolve failed"). Results for these products may differ on other machines.
- Web search often failed. No search API key was used. DuckDuckGo blocked most automated searches (HTTP 202), and on 21 October Bing often returned nothing relevant. Searches with no usable results: Resend 6 of 6, Trigger.dev 7 of 7, Postmark 9 of 10, WorkOS 12 of 16, Supabase 15 of 23, Auth0 6 of 9, SendGrid 3 of 4, QStash 7 of 15 (Neon 2 of 11, Prisma 1 of 13, Clerk 2 of 16). Agents then opened docs directly. This may have hidden no-account options and makes "found the no-account option" less reliable for those products.
- Analyst mistakes we found (code numbers left unchanged, flagged instead):
- WorkOS: a run that met the definition was marked unsuccessful.
- Prisma: the analyst said the quickstart has no pointer to the no-signup option, but that page shows
npx create-db@latest(Footprint's keyword check missed "create-db").
- Footprint bugs found (not fixed during the benchmark, because they did not block tests; all five fixed in Stage 1 on 22 Oct 2026, see CHANGELOG.md, and the message files in
reports/re-checked by code):- Message files can contain unfilled number placeholders (Prisma: "…{real:all} runs read back a row…"; Supabase: "{shortcut:B} runs found…").
- Numbers in the X note are not checked by code (Prisma's says "worked in 3 of 4 hinted runs"; code counted 0 of 4).
- The quote check removed Clerk's well-supported problem.
- One rate-limit answer is enough to switch analysts (Resend's report was written by GLM; QStash's GLM report failed).
- Runs that hang on the network end as "time limit" instead of "environment problem".
- Budget: four tests went over 2 million tokens: Prisma Postgres (attempt 1), WorkOS, Inngest (attempt 1) and QStash's re-run (2,049,980 after the approved re-analysis). The owner approved continuing each time. QStash's two attempts total 2,902,089 tokens.
- Small samples: 4 to 8 runs per product. A single run changes a rate a lot.
- Cheaper agents: both agents are "Flash" models; stronger agents may do better.
- No testing logic was changed during this benchmark. The only special setting was Footprint's own "one run at a time" option for the approved re-runs (the agents got the same instructions).
8. Cost and time
| Product | Result used | Tokens | Minutes | Extra attempts (tokens, minutes) |
|---|---|---|---|---|
| Upstash QStash | re-run, analysed again | 2,049,980 | 16.6 | attempt 1: 852,109, 23.0 |
| Neon | attempt 1 | 353,906 | 3.9 | none |
| Prisma Postgres | attempt 1 | 2,114,316 | 31.1 | cut-off re-run: 205,264, 14.8; re-run: 435,349, 56.8 |
| Auth0 | attempt 1 | 1,081,604 | 12.2 | none |
| Clerk | attempt 1 | 1,270,303 | 17.0 | none |
| Postmark | attempt 1 | 611,182 | 4.5 | none |
| Resend | attempt 1 | 270,076 | 7.9 | none |
| SendGrid | attempt 1 | 185,727 | 1.5 | none |
| Supabase | attempt 1 | 861,272 | 19.2 | none |
| Trigger.dev | attempt 1 | 733,886 | 12.8 | none |
| WorkOS | attempt 1 | 2,047,127 | 22.0 | none |
| Inngest (skipped) | not ranked | none | none | attempt 1: 2,006,043, 11.0; cancelled re-run: 690,733, 9.4 |
| Total | 11,579,379 | 148.7 | 4,189,498 tokens, 115.0 minutes |
- Grand total: 15,768,877 tokens and 263.7 minutes of test time (about 4 hours 24 minutes).
- Minutes are each test's own running time. The tests ran one product at a time, from 20 October 19:49 to 21 October about 11:20 (local time), with an overnight break.
- River.ai does not report prices, so no money cost is shown.