Benchmark report

Task verified

12m 14s elapsed

The saved attempt met the documented success criterion.

Experience score

98/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • SDocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs S, Current SAInstallSAuthSExecuteSDocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Only the PEP 668 externally-managed Python environment, which rejected pip3 install and forced a venv; everything Tavily-side worked on the first try.

Where the time went

12:14 agent time
  • Starting3:12
  • Research1:42
  • Attempt4:15
  • Verify3:05
12m 14sTotal elapsed
1:20Time to first success
0:00Waited on the owner

Judge reason

Saved code (tavily-map/map_postgres_docs.py) makes exactly one live POST https://api.tavily.com/map with Bearer auth and body {url: https://www.postgresql.org/docs/17/, instructions: "tutorial", max_depth 3, limit 200, select_paths ^/docs/17/.*}; the fetched official reference confirms `instructions` is the documented natural-language filter and carries no beta/preview/gating marking. Transcript line 33-36 shows the script actually executed (requests 2.34.2 in a venv, key read from the benchmark-provided private file), returning request_id eacb41c6-e572-44b6-b4ce-52163b77bd72, response_time 13.01s, usage {credits: 3}. map_response.json holds 18 URLs; I re-verified all 18 start with https://www.postgresql.org/docs/17/, that run_output.txt's printed count (18) and exactly the first 10 URLs match the response, and that the only outbound HTTP call in the code is the single map POST (no per-page fetching, no local filtering - list is printed straight from data['results']). Nothing mocked or hard-coded.

Attempt record

  1. 7:19Answered from saved key: Tavily API keyno wait
  2. —First success verified

Attempt artifacts saved.