Benchmark report

Task verified

7m 30s elapsed

The saved attempt met the documented success criterion.

Experience score

86/ 100

Agent-Ready
  • AInstall
  • AAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth A, Execute S, Docs A, Current SAInstallAAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Two small stumbles: PEP 668 blocking the first pip install, then the free-plan org_id 400 on the first management-API call.

Where the time went

7:30 agent time
  • Starting0:01
  • Research0:19
  • Attempt5:21
  • Verify1:49
7m 30sTotal elapsed
1:59Time to first success
0:00Waited on the owner

Judge reason

Real end-to-end run verified from saved code and outputs. neon_phase1.py calls POST https://console.neon.tech/api/v2/projects (HTTP 201, artifacts/01, project tpcbench-5aaf3a43da / hidden-pond-45080216), connects with psycopg 3.3.8 using sslmode=verify-full&sslrootcert=system (artifacts/02 saved conninfo, redacted password), and runs CREATE TABLE items (id integer PRIMARY KEY, name text NOT NULL), three INSERT INTO items (id,name) VALUES (%s,%s) with bound tuples, and SELECT ... WHERE id >= %s ORDER BY id ASC with parameter 2 - no concatenation. Printed rows are exactly (2,'beta') then (3,'gamma') (artifacts/03 and phase1_console.txt, matching transcript tool output). TLS is further proven by a negative control: identical connect with an untrusted self-signed CA fails with 'SSL error: certificate verify failed' (artifacts/08); pg_stat_ssl ssl=f is expected behind the pooler. Cleanup: DELETE /projects/hidden-pond-45080216 HTTP 200 with full project body (artifacts/04), follow-up GET HTTP 404 'project not found' (artifacts/05), id absent from the org list (artifacts/06), SQL endpoint unreachable after proxy cache expiry (artifacts/09), and no tpcbench- project remains (artifacts/10). Nothing mocked. Flaw noted: the first redactor missed JSON password fields, so the role password was briefly written to artifacts/01 and printed once by the agent's own grep in the transcript; the agent caught it, scrubbed the file and hardened the redactor, the project (and thus the credential) was then deleted, and my independent grep over all saved code and artifacts finds no password, token, or password-bearing connection string.

Attempt record

  1. 0:55Answered from saved key: NEON_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.