Benchmark report
Task verified
12m 14s elapsed
The saved attempt met the documented success criterion.
98/ 100
Agent-Ready- AInstall
- SAuth
- SExecute
- SDocs
- SCurrent
Only the PEP 668 externally-managed Python environment, which rejected pip3 install and forced a venv; everything Tavily-side worked on the first try.
Where the time went
12:14 agent time- Starting3:12
- Research1:42
- Attempt4:15
- Verify3:05
Judge reason
Saved code (tavily-map/map_postgres_docs.py) makes exactly one live POST https://api.tavily.com/map with Bearer auth and body {url: https://www.postgresql.org/docs/17/, instructions: "tutorial", max_depth 3, limit 200, select_paths ^/docs/17/.*}; the fetched official reference confirms `instructions` is the documented natural-language filter and carries no beta/preview/gating marking. Transcript line 33-36 shows the script actually executed (requests 2.34.2 in a venv, key read from the benchmark-provided private file), returning request_id eacb41c6-e572-44b6-b4ce-52163b77bd72, response_time 13.01s, usage {credits: 3}. map_response.json holds 18 URLs; I re-verified all 18 start with https://www.postgresql.org/docs/17/, that run_output.txt's printed count (18) and exactly the first 10 URLs match the response, and that the only outbound HTTP call in the code is the single map POST (no per-page fetching, no local filtering - list is printed straight from data['results']). Nothing mocked or hard-coded.
Attempt record
- 7:19Answered from saved key: Tavily API keyno wait
- —First success verified
Attempt artifacts saved.