browser-use agents on cheap models fabricate results; validate against visited URLs

Observed behaviour

On 7 September 2026, browser-use 0.13.10 was run against an online marketplace through OpenRouter.

• google/gemini-3.5-flash-lite called done after a single search step with plausible-looking but invented item ids, prices and ratings. Nothing in the task prompt stopped it.
• google/gemini-3.7-flash (about 331k tokens, cents per run) visited real item pages and returned real ids, but still misread one variant price: it reported NZ$41.10 where the page showed NZ$52.29 for the selected variant.

Defences that worked

• Task rules stating "never invent an item id, price or rating" and "do not call done before visiting at least N item pages", with the exact URL pattern the answer must contain.
• Post-run validation: extract ids from history.urls() and flag any result whose URL was never visited. A wrapper that returns the visited URLs next to the result makes the comparison easy.
• Spot-checking every number that matters (price after variant selection, stock, dates) with a real page load.
• A Pydantic output_model_schema, so the result is structured enough to validate mechanically.

Environment gotchas

• A scratch file named backoff.py in the working directory shadows the backoff package that posthog imports, which breaks import browser_use. Running from a clean directory avoids it.
• Setting ANONYMIZED_TELEMETRY=false and BROWSER_USE_CLOUD_SYNC=false disables telemetry and cloud sync.
• browser_use.__version__ does not exist; uv pip show browser-use reports the version.