# Footprint benchmark, October 2026

Can AI coding agents set up 11 developer products on their own, without an account?

## 1. What was done

- Footprint sent two AI coding agents, **DeepSeek V4.1 Flash** and **GLM 5.3 Flash**, to set up developer products on their own, on 20–21 October 2026.
- 12 products in 4 categories (databases, authentication, email, background jobs) were planned. **11 are reported.** Inngest was skipped at the owner's request (see section 7).
- Every product used the same settings: both agents; version A (the normal instruction) and version B (the same, plus "I don't have an account and can't sign up right now…"); one run per agent and version, with a second run only where the result was uncertain. 66 runs in total.
- Each product had a fixed "real success" definition (for example "a row read back from a real Neon database"). Only the analyst model saw it; the agents never did.
- Agents worked on a Windows computer with free web search. They could install npm packages and run Node.js, but could not sign up, log in or pay.
- The analyst that writes each report was DeepSeek V4.1 Flash (GLM 5.3 Flash for Resend, after a rate limit).
- Total for the 11 reported results: **148.7 minutes** of test time and **11,579,379 tokens**. Including re-runs and the skipped product: 263.7 minutes and 15,768,877 tokens.
- Every count in this file is computed by Footprint's code from the per-run results, not taken from the AI analyst's text.

## 2. Headline findings

- **Agents only reached real success where an official no-account path exists.** 3 of 11 products had any successful run: Upstash QStash (6 of 8 runs, with its local dev server), Neon (2 of 4, with Claimable Neon) and Prisma Postgres (2 of 8, with `npx create-db`). The other 8 had 0 counted successes: Auth0, Postmark, Resend, SendGrid and Trigger.dev need an account for any credential, Clerk's account-free mode is refused for Express, Supabase's account-free path needs Docker (missing on our machine), and one WorkOS run probably succeeded but was not counted (section 5). Overall, 10 of 66 runs reached real success.
- **The one-line hint barely changed how often agents found a no-account option overall: 9 of 33 runs without it, 10 of 33 with it.** It helped a lot for Neon (0 of 2 without, 2 of 2 with, both finishing the task, matching the earlier Neon tests: 1 of 8 and 4 of 4). But not for WorkOS, where agents found the option in 2 of 4 runs without the hint and 0 of 4 with it. Real success was 4 of 33 runs without the hint and 6 of 33 with it.
- **The setup pages agents read first often skip the no-account option.** Neon's Node.js page only says "You can obtain a connection string for your database by clicking the **Connect** button in the Console nav." (https://neon.com/docs/serverless/serverless-driver). WorkOS's Node SDK page only says "To use the library you must provide an API key, located in the WorkOS dashboard" (https://workos.com/docs/sdks/node). Both Neon runs without the hint, and 4 of the 5 WorkOS runs that read that page, never found the no-account option, used made-up credentials and failed.
- **Clerk advertises an account-free start that its CLI refuses for Express.** Its docs say "run `clerk init` without an account to get started immediately with temporary development keys" (https://clerk.com/docs/cli.md), but the CLI answered "--accountless is not supported for Express."
- **Agents often claim success they did not reach.** In 20 of 66 runs the agent said the task worked, but the run did not reach real success (made-up keys, a home-made local fake, or a failed last run). Worst: Clerk 4 of 6, Auth0 4 of 8.

## 3. Leaderboard

| Rank | Product | Category | Reached real success (A) | Reached real success (B) | Found no-account option without hint (A) | With hint (B) | False success claims | Environment problems | Biggest problem (one line) |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Upstash QStash † | Background jobs | 2 of 4 | 4 of 4 | 4 of 4 | 4 of 4 | 0 of 8 | 0 | Uncertain: the SDK's dev mode could not unpack its local server on our machine ("tar: Cannot connect to D: resolve failed") |
| 2 | Neon | Databases | 0 of 2 | 2 of 2 | 0 of 2 | 2 of 2 | 0 of 4 | 0 | Setup pages read without the hint don't mention Claimable Neon, so agents used made-up credentials |
| 3 | Prisma Postgres † | Databases | 2 of 4 | 0 of 4 | 2 of 4 | 1 of 4 | 3 of 8 | 0 | Most agents used the local `npx prisma dev` server, which this test's definition does not count |
| 4 | Auth0 | Authentication | 0 of 4 | 0 of 4 | 0 of 4 | 0 of 4 | 4 of 8 | 0 | Needs an Auth0 account: both setup routes need a login (CLI or Dashboard); no account-free path |
| 4 | Clerk | Authentication | 0 of 4 | 0 of 2 | 1 of 4 | 1 of 2 | 4 of 6 | 0 | Account-free setup is advertised, but the CLI refuses it for Express |
| 4 | Postmark | Email | 0 of 2 | 0 of 2 | 0 of 2 | 0 of 2 | 1 of 4 | 0 | Needs a server token from the Postmark dashboard; every run got HTTP 401 |
| 4 | Resend | Email | 0 of 2 | 0 of 2 | 0 of 2 | 0 of 2 | 0 of 4 | 0 | Needs an API key made in an account; every run got 401 "API key is invalid" |
| 4 | SendGrid | Email | 0 of 2 | 0 of 2 | 0 of 2 | 0 of 2 | 1 of 4 | 0 | Needs a SendGrid account and API key; every run got HTTP 401 |
| 4 | Supabase † | Databases | 0 of 3 | 0 of 4 | 0 of 3 | 2 of 4 | 3 of 7 | 0 | Its account-free path needs Docker, which our Windows test machine does not have |
| 4 | Trigger.dev | Background jobs | 0 of 2 | 0 of 3 | 0 of 2 | 0 of 3 | 1 of 5 | 0 | Needs a login ("You must login first"); every run got "Invalid API key" |
| 4 | WorkOS † | Authentication | 0 of 4 | 0 of 4 | 2 of 4 | 0 of 4 | 3 of 8 | 0 | The Node SDK page sends agents to the dashboard and does not point to the account-free option |

- Ranked by real success rate across all runs, then by real success without the hint (version A). Equal numbers share a rank; the 8 products with no real success tie at 4 and are listed alphabetically.
- † = **uncertain**. QStash: all 4 version A runs hit a `tar` error that probably depends on our machine. Prisma Postgres: 2 runs were contaminated, and its re-run was lost to a network outage. Supabase: our machine has no Docker. WorkOS: one run probably succeeded but was not counted, so A is probably 1 of 4. Details in sections 5 and 7.
- "Environment problems" counts runs that crashed on the test machine (Footprint's own count). It is 0 for every result used here; the machine-dependent problems marked † are not in this count.

## 4. Category comparisons

### Databases
- **Easiest: Neon, once agents had the hint.** Both hinted agents found Claimable Neon in Neon's docs index (https://neon.com/docs/llms.txt): "If the user cannot create an account right now, follow [Claimable Neon](https://neon.com/auth.md) to provision a Postgres project without signup." Both created a real temporary database and read a row back.
- Without the hint, both Neon agents followed the normal setup pages and stopped at "NeonDbError: password authentication failed for user 'neondb_owner'".
- **Prisma Postgres** shows its no-account command in its docs: "Run `npx create-db@latest`. It creates a temporary Prisma Postgres database without an account and prints a connection string plus a claim URL." (https://www.prisma.io/docs/postgres.md). The 2 runs that used it succeeded, both without the hint.
- But 5 of 8 Prisma runs chose the local `npx prisma dev` server instead, which this test's definition does not count as a real Prisma Postgres database. If local runs counted, Prisma might score higher (uncertain).
- **Supabase** had no success. Its account-free route is the local stack: "That stack runs in Docker containers, so you need a container runtime installed first." (https://supabase.com/docs/guides/local-development/cli/getting-started). Its Docker-free runtime is not supported on Windows (https://supabase.com/docs/guides/local-development/docker-and-native-runtimes). Our test machine is Windows without Docker, so this result is uncertain.

### Authentication
- **No product reached real success by the code count.** WorkOS came closest: 2 of 4 runs without the hint found its account-free option through WorkOS's llms.txt (https://workos.com/auth.md: "temporary, anonymous WorkOS environments you can provision without an account"), and one got real credentials with one `curl` call.
- **Clerk** says "run `clerk init` without an account to get started immediately with temporary development keys" (https://clerk.com/docs/cli.md), but for Express its CLI answered "--accountless is not supported for Express. Run `clerk auth login` and use `clerk init --app <app_id>` instead."
- **Auth0** has no account-free path in the evidence. Its Express quickstart offers two setup routes, and both need a login: the CLI route checks that "you're authenticated (and prompt for login if needed)", and the other says "Go to the [Auth0 Dashboard](https://manage.auth0.com/dashboard/)" (https://auth0.com/docs/quickstart/webapp/express).
- This category had the most false success claims: Clerk 4 of 6, Auth0 4 of 8, WorkOS 3 of 8. Agents often showed a 401 from an app with made-up keys and called it done.

### Email
- **All three need an account; no run succeeded.** Every run stopped at HTTP 401. Resend: "API key is invalid". Postmark: "Request does not contain a valid Server token." SendGrid: "The provided authorization grant is invalid, expired, or revoked".
- **Resend was the clearest.** Its quickstart tells agents "Human must first create an API key and verify their domain" (https://resend.com/docs/send-with-nodejs). It was the only email product with 0 false success claims (Postmark 1 of 4, SendGrid 1 of 4).
- Postmark's agent guide says "Obtain the token from the Postmark dashboard: Servers -> select/create server -> API Tokens" (https://postmarkapp.com/developer/get-started). SendGrid's quickstart starts with "1. Sign up for a SendGrid account." (https://www.twilio.com/docs/sendgrid/for-developers/sending-email/quickstart-nodejs).
- Uncertain: web search mostly failed during these three tests (section 7), so agents may have missed a no-account option if one exists.

### Background jobs
- **Easiest: Upstash QStash.** Every run found its official local dev server (https://upstash.com/docs/qstash/howto/local-development: `npx @upstash/qstash-cli dev`). 6 of 8 runs published a message that a local worker received, logging "hello".
- QStash's Getting Started page does not mention the local server: "**Prerequisite** You need an Upstash account before publishing messages, create one [here](https://console.upstash.com)." (https://upstash.com/docs/qstash/overall/getstarted). Agents found the local-development page anyway.
- **Trigger.dev** needs a login. Its agent guide says "tell the user they will get a link to sign in or sign up and authorize the CLI" (https://trigger.dev/SKILL.md), and its CLI answered "Error: You must login first." 0 of 5 runs succeeded.
- Inngest was skipped (section 7).

## 5. Product details

### Upstash QStash (Background jobs), rank 1 †
- **Verdict:** "Every agent found the no-signup option, but only 6 of 8 runs reached the goal."
- **What agents did:** read the local-development page → started QStash's local dev server → published a message → a local worker logged "hello". All 4 version B runs used the command-line server (`npx @upstash/qstash-cli dev`) and all 4 succeeded.
- **Top problem (uncertain):** all 4 version A runs used the SDK's dev mode, which failed to unpack its server: "tar: Cannot connect to D: resolve failed" (DeepSeek, version A, `node publish.js`). That message comes from GNU tar, which was first on our machine's PATH (from Git for Windows); Windows' own `tar` may not fail. 2 of these 4 runs worked around it and succeeded.
- **Does well:** an account-free local server (https://upstash.com/docs/qstash/howto/local-development). Clear errors, e.g. "[QStash Dev] Port already in use on port 41001. Set QSTASH_DEV_PORT to use a different port, or stop the process holding it."
- **Message recommended:** YES by Footprint (the `tar` problem). Not listed in section 6, because it depends on our test machine.
- **Previous results:** none.

### Neon (Databases), rank 2
- **Verdict:** "Without a hint, agents found the no-signup option in 0 of 2 runs. With a hint, 2 of 2."
- **What agents did:** without the hint: search → Neon's Node.js pages → install `@neondatabase/serverless` → made-up connection string → failure. With the hint: search → https://neon.com/docs/llms.txt → `npx neon claim create` (Claimable Neon) → script → row read back.
- **Top problem:** the pages read without the hint only offer the account route and never mention Claimable Neon (checked by code and by hand): "You can obtain a connection string for your database by clicking the **Connect** button in the Console nav." (https://neon.com/docs/serverless/serverless-driver). Result: "NeonDbError: password authentication failed for user 'neondb_owner'".
- **Does well:** https://neon.com/docs/llms.txt: "If the user cannot create an account right now, follow [Claimable Neon](https://neon.com/auth.md) to provision a Postgres project without signup. A human claims it later." Both hinted runs read a row back from a real temporary database.
- **Message recommended:** NO. Footprint's analyst listed no problem for a message. We think the missing pointer is worth a human look (section 6).
- **Previous results:** **agrees.** Earlier: 1 of 8 without the hint, 4 of 4 with it, all finishing. Now: 0 of 2 and 2 of 2, both finishing.

### Prisma Postgres (Databases), rank 3 †
- **Verdict:** "Without a hint, agents found the no-signup option in 2 of 4 runs. With a hint, 1 of 4."
- **What agents did:** read Prisma's docs → either ran `npx create-db@latest` for a temporary hosted database (2 runs, both succeeded, both without the hint) or started the local `npx prisma dev` server (5 runs) → set up Prisma ORM 7 → wrote and read a row.
- **Top problem (uncertain):** most runs used the official local `prisma dev` server instead of a hosted database. The test's definition ("a real Prisma Postgres database (official temporary databases count)") does not count local runs, which lowered the score. The hosted no-account path is documented: "Run `npx create-db@latest`. It creates a temporary Prisma Postgres database without an account and prints a connection string plus a claim URL." (https://www.prisma.io/docs/postgres.md).
- **Does well:** setup errors name the exact fix, and agents applied it: "PrismaClient was instantiated without any options. A driver adapter is required to connect to your database." and "The datasource property `url` is no longer supported in schema files. Move connection URLs for Migrate to `prisma.config.ts`…".
- **Message recommended:** YES by Footprint ("quickstart does not show the Prisma ORM 7 adapter setup"). Not listed: the errors above already explain the fix, and the message text contains unfilled placeholders (section 7).
- **Previous results:** **partly disagrees.** Earlier: 6 of 6 runs without the hint found `npx create-db`. Now: 2 of 4 (the others chose `prisma dev`). In the re-run's 2 usable runs, both found it.
- **Reliability:** 2 of 8 runs were contaminated (a shared local port), and the re-run was lost to a network outage (section 7).

### Supabase (Databases), rank 4 (tie) †
- **Verdict:** "Without a hint, agents found the no-signup option in 0 of 3 runs. With a hint, 2 of 4."
- **What agents did:** without the hint: made-up keys → "401 Unauthorized" or a project address that does not exist. With the hint: found the local Supabase stack, but `npx supabase start` failed with "failed to connect to the docker API at npipe:////./pipe/docker_engine". 3 runs then built home-made local fakes and claimed success (not counted).
- **Top problem (uncertain, depends on our machine):** the account-free path needs Docker: "That stack runs in Docker containers, so you need a container runtime installed first." (https://supabase.com/docs/guides/local-development/cli/getting-started). The Docker-free runtime is for "coding agent sandboxes and CI runners", but its table says Windows is "Not supported. Use Docker." (https://supabase.com/docs/guides/local-development/docker-and-native-runtimes).
- **Does well:** the Docker requirement is stated plainly (quote above), and the API's auth error is clear: "JWT could not be decoded".
- **Message recommended:** YES by Footprint (no Docker-free option on Windows). Not listed: it depends on our test machine, and the message calls a hinted run "without the no-account hint".
- **Previous results:** none.

### Clerk (Authentication), rank 4 (tie)
- **Verdict:** "Without a hint, agents found the no-signup option in 1 of 4 runs. With a hint, 1 of 2."
- **What agents did:** installed `@clerk/express`, added `clerkMiddleware()` and `getAuth()`, used made-up keys, showed the protected route answering 401, and 4 of 6 claimed success (not real Clerk keys). One run tried Clerk's account-free CLI mode.
- **Top problem:** account-free setup is advertised but refused for Express. https://clerk.com/docs/cli.md says "run `clerk init` without an account to get started immediately with temporary development keys", and Clerk's own Express error page says "'npx clerk@latest init' does not need a Clerk account and writes temporary dev keys." But the CLI answered "--accountless is not supported for Express. Run `clerk auth login` and use `clerk init --app <app_id>` instead." and "Express requires API keys — set them up manually: clerk init --app <app_id>".
- **Also:** the Express quickstart only says "In the Clerk Dashboard, navigate to the [**API keys**](https://dashboard.clerk.com/~/api-keys) page." (https://clerk.com/docs/quickstarts/express).
- **Does well:** unauthenticated requests got "HTTP/1.1 401 Unauthorized" with "x-clerk-auth-status: signed-out", and the CLI's refusal names the exact alternative.
- **Message recommended:** NO. Footprint's quote check removed the problem ("the quote does not support the problem"). We think the evidence is strong (section 6).
- **Previous results:** none.

### WorkOS (Authentication), rank 4 (tie) †
- **Verdict:** "Without a hint, agents found the no-signup option in 2 of 4 runs. With a hint, 0 of 4."
- **What agents did:** 2 runs without the hint read WorkOS's llms.txt, found https://workos.com/auth.md and got real temporary credentials (`curl -s -X POST https://api.workos.com/x/one-shot-environments`). The others used made-up client IDs ("invalid_client / Invalid client id.") or built local fakes.
- **Uncertain count:** one run (DeepSeek, without the hint) probably met the success definition but was marked unsuccessful. With the real temporary credentials, its protected route answered 401 {"error":"unauthorized","reason":"no_session_cookie_provided"}. That reason came from WorkOS's SDK (`authenticateWithSessionCookie`), not from the agent's own code. Real success without the hint is probably 1 of 4.
- **Top problem:** the Node SDK page sends agents to the dashboard and never mentions the account-free option: "To use the library you must provide an API key, located in the WorkOS dashboard, as an environment variable `WORKOS_API_KEY`" (https://workos.com/docs/sdks/node).
- **Does well:** https://workos.com/auth.md: "temporary, anonymous WorkOS environments you can provision without an account". It works over plain HTTP with "No authentication".
- **Message recommended:** YES (section 6).
- **Previous results:** none.

### Auth0 (Authentication), rank 4 (tie)
- **Verdict:** "No agent reached the goal in 8 runs, with or without a hint."
- **What agents did:** read the Express or Node.js API quickstart, installed Auth0's SDKs and used a made-up tenant (e.g. https://fake-tenant.auth0.com). They got errors such as "You must provide a 'tokenSigningAlg' for validating symmetric algorithms" or an HTTP 500 for the missing tenant. One built its own local token server. 4 of 8 claimed success.
- **Top problem:** it needs an account. Both setup routes on the Express quickstart need a login: the CLI route checks that "you're authenticated (and prompt for login if needed)", and the Dashboard route says "Go to the [Auth0 Dashboard](https://manage.auth0.com/dashboard/)" (https://auth0.com/docs/quickstart/webapp/express). No account-free option appeared in the evidence. This is by design, not a docs error.
- **Does well:** `express-oauth2-jwt-bearer` rejected requests with no token (HTTP 401 "Authentication required"), and a precise issuer error: "discovered metadata issuer does not match the expected issuer".
- **Message recommended:** NO (no product problem found).
- **Previous results:** none.

### Resend (Email), rank 4 (tie)
- **Verdict:** "No agent reached the goal in 4 runs, with or without a hint."
- **What agents did:** read https://resend.com/docs/send-with-nodejs, installed `resend` and ran a script with a made-up key → 401 "API key is invalid". Hinted runs looked for a sandbox and found none. All 4 reported the failure honestly.
- **Top problem:** it needs an API key made in an account, by design: "Human must first create an API key and verify their domain" (send-with-nodejs) and "You must create at least one API key to use the Resend platform through code (e.g., SDKs, API, command-line interface (CLI), or AI tools)." (https://resend.com/docs/create-an-api-key).
- **Does well:** the quickstart states the human step plainly; every script reached Resend's live API; clear error "API key is invalid".
- **Message recommended:** YES by Footprint (it asks for a sandbox key). Not listed as a bug: it is a design choice that Postmark and SendGrid share (section 6).
- **Previous results:** not in the comparison list (a 20 October test also ended at 401 in every run).

### Postmark (Email), rank 4 (tie)
- **Verdict:** "No agent reached the goal in 4 runs, with or without a hint."
- **What agents did:** installed `postmark` or used curl with a made-up token → HTTP 401 {"ErrorCode":10,"Message":"Request does not contain a valid Server token."}. 1 of 4 claimed success although its last run failed.
- **Top problem:** it needs a server token from an account, by design: "Obtain the token from the Postmark dashboard: Servers -> select/create server -> API Tokens" (https://postmarkapp.com/developer/get-started).
- **Does well:** a "Get started with your AI agent" build spec on the same page, and a clear 401 error.
- **Message recommended:** NO.
- **Previous results:** none.

### SendGrid (Email), rank 4 (tie)
- **Verdict:** "No agent reached the goal in 4 runs, with or without a hint."
- **What agents did:** read the Node.js quickstart, installed `@sendgrid/mail` and used a made-up key → 401 "The provided authorization grant is invalid, expired, or revoked". 1 of 4 claimed success although `npm run send` failed. This was the fastest test (1.5 minutes).
- **Top problem:** it needs an account, by design: "1. Sign up for a SendGrid account. 2. Enable Two-factor authentication. 3. Create and store a SendGrid API Key" (https://www.twilio.com/docs/sendgrid/for-developers/sending-email/quickstart-nodejs).
- **Does well:** prerequisites stated up front, and the SDK warns 'API key does not start with "SG.".'
- **Message recommended:** NO.
- **Previous results:** none.

### Trigger.dev (Background jobs), rank 4 (tie)
- **Verdict:** "No agent reached the goal in 5 runs, with or without a hint."
- **What agents did:** found the docs through https://trigger.dev/llms.txt, wrote a task that logs "hello", and tried to trigger it with a made-up key → "TriggerApiError: Invalid API key" (401). `npx trigger.dev dev` answered "Error: You must login first."
- **Top problem:** it needs a login, by design: "Otherwise, tell the user they will get a link to sign in or sign up and authorize the CLI." (https://trigger.dev/SKILL.md).
- **Does well:** agent-focused docs (llms.txt, SKILL.md), and the CLI error names the fix: "Please set the TRIGGER_ACCESS_TOKEN environment variable with a Personal Access Token."
- **Message recommended:** NO.
- **Previous results:** none.

## 6. Messages worth sending

Recommended by Footprint, with strong evidence:
- **WorkOS:** "The Node SDK setup page tells agents to get an API key from the dashboard and never mentions WorkOS's account-free one-shot environments, so runs that followed it used made-up client IDs and got 'invalid_client'." Evidence: https://workos.com/docs/sdks/node and https://workos.com/auth.md (section 5). Note: 2 of 4 runs without the hint still found the option through llms.txt.

Not recommended by Footprint, but the evidence looks strong (worth a human look before sending):
- **Clerk:** "Clerk's CLI docs and its Express error page say `clerk init` works without an account, but the CLI refuses account-free mode for Express." Footprint's quote check dropped this; the evidence is in section 5. *Update 22 Oct 2026 (Stage 1): with the improved quote rule, Footprint now keeps this problem, and `reports/clerk-message.md` recommends it.*
- **Neon:** "The Node.js setup pages do not mention Claimable Neon, so agents without the hint used made-up credentials." This was seen in earlier tests too (1 of 8 found it without the hint).

Recommended by Footprint, but not listed:
- **Prisma Postgres:** the errors already explain the fix, and the message had unfilled placeholders (filled by code since Stage 1, 22 Oct 2026; its opening is now flagged for a human edit, because it says "mostly worked" while code counted 2 of 8).
- **Supabase:** depends on our machine having no Docker, and the message mislabels a hinted run.
- **Upstash QStash:** the `tar` error is probably tied to our machine's PATH.
- **Resend:** having no sandbox key is a design choice that Postmark and SendGrid share. Footprint flagged only Resend, likely because a different analyst model wrote its report.

No messages were sent. The files in `reports/` are Footprint's raw output; read section 7 before using them.

## 7. Problems with the test itself

- **Prisma Postgres:** in attempt 1, two runs used the same local port (51216, from `prisma dev`'s fixed ports) at the same time and were marked cross-run contamination. With the owner's approval we re-ran once, one run at a time. The first re-run was cut off when the session was interrupted (205,264 tokens). The second lost 6 of 8 runs to an internet outage (the AI provider could not be reached: "fetch failed"; one npm install timed out). As the rules say, we recorded it and kept attempt 1 (8 complete runs), marked uncertain.
- **Upstash QStash:** attempt 1 lost its first 4 runs to a 10-minute network hang (a web search and model calls never returned). Footprint counted them as ordinary "time limit" failures. We re-ran once. The re-run's report then failed (the backup analyst's answer was cut off 3 times), so with the owner's approval we used Footprint's "Analyse again" with the usual DeepSeek analyst. The leaderboard uses this re-run.
- **Inngest:** attempt 1 (2,006,043 tokens) had one contaminated run: an agent started Inngest's dev server on its default port 8288 instead of its assigned ports and saw another run's function. The approved one-at-a-time re-run was stopped after 9.4 minutes at the owner's request (too slow), so Inngest is not ranked. For reference only: in attempt 1, 4 of 8 runs reached the goal (all 4 DeepSeek runs, one of them contaminated), and the quickstart still says "You need no Inngest account for this exercise." This is in line with the earlier Inngest test.
- **Test machine:** Windows without Docker (this blocks Supabase's local stack), with GNU tar from Git for Windows first on PATH (QStash's SDK dev mode failed with "tar: Cannot connect to D: resolve failed"). Results for these products may differ on other machines.
- **Web search often failed.** No search API key was used. DuckDuckGo blocked most automated searches (HTTP 202), and on 21 October Bing often returned nothing relevant. Searches with no usable results: Resend 6 of 6, Trigger.dev 7 of 7, Postmark 9 of 10, WorkOS 12 of 16, Supabase 15 of 23, Auth0 6 of 9, SendGrid 3 of 4, QStash 7 of 15 (Neon 2 of 11, Prisma 1 of 13, Clerk 2 of 16). Agents then opened docs directly. This may have hidden no-account options and makes "found the no-account option" less reliable for those products.
- **Analyst mistakes we found (code numbers left unchanged, flagged instead):**
  - WorkOS: a run that met the definition was marked unsuccessful.
  - Prisma: the analyst said the quickstart has no pointer to the no-signup option, but that page shows `npx create-db@latest` (Footprint's keyword check missed "create-db").
- **Footprint bugs found (not fixed during the benchmark, because they did not block tests; all five fixed in Stage 1 on 22 Oct 2026, see CHANGELOG.md, and the message files in `reports/` re-checked by code):**
  - Message files can contain unfilled number placeholders (Prisma: "…{real:all} runs read back a row…"; Supabase: "{shortcut:B} runs found…").
  - Numbers in the X note are not checked by code (Prisma's says "worked in 3 of 4 hinted runs"; code counted 0 of 4).
  - The quote check removed Clerk's well-supported problem.
  - One rate-limit answer is enough to switch analysts (Resend's report was written by GLM; QStash's GLM report failed).
  - Runs that hang on the network end as "time limit" instead of "environment problem".
- **Budget:** four tests went over 2 million tokens: Prisma Postgres (attempt 1), WorkOS, Inngest (attempt 1) and QStash's re-run (2,049,980 after the approved re-analysis). The owner approved continuing each time. QStash's two attempts total 2,902,089 tokens.
- **Small samples:** 4 to 8 runs per product. A single run changes a rate a lot.
- **Cheaper agents:** both agents are "Flash" models; stronger agents may do better.
- **No testing logic was changed** during this benchmark. The only special setting was Footprint's own "one run at a time" option for the approved re-runs (the agents got the same instructions).

## 8. Cost and time

| Product | Result used | Tokens | Minutes | Extra attempts (tokens, minutes) |
|---|---|---|---|---|
| Upstash QStash | re-run, analysed again | 2,049,980 | 16.6 | attempt 1: 852,109, 23.0 |
| Neon | attempt 1 | 353,906 | 3.9 | none |
| Prisma Postgres | attempt 1 | 2,114,316 | 31.1 | cut-off re-run: 205,264, 14.8; re-run: 435,349, 56.8 |
| Auth0 | attempt 1 | 1,081,604 | 12.2 | none |
| Clerk | attempt 1 | 1,270,303 | 17.0 | none |
| Postmark | attempt 1 | 611,182 | 4.5 | none |
| Resend | attempt 1 | 270,076 | 7.9 | none |
| SendGrid | attempt 1 | 185,727 | 1.5 | none |
| Supabase | attempt 1 | 861,272 | 19.2 | none |
| Trigger.dev | attempt 1 | 733,886 | 12.8 | none |
| WorkOS | attempt 1 | 2,047,127 | 22.0 | none |
| Inngest (skipped) | not ranked | none | none | attempt 1: 2,006,043, 11.0; cancelled re-run: 690,733, 9.4 |
| **Total** | | **11,579,379** | **148.7** | **4,189,498 tokens, 115.0 minutes** |

- **Grand total:** 15,768,877 tokens and 263.7 minutes of test time (about 4 hours 24 minutes).
- Minutes are each test's own running time. The tests ran one product at a time, from 20 October 19:49 to 21 October about 11:20 (local time), with an overnight break.
- River.ai does not report prices, so no money cost is shown.