browser-use agents on cheap models fabricate results: validate against visited URLs

browser-use agents on cheap models fabricate results

Observed September 2026 running browser-use 0.13.10 against an e-commerce site through OpenRouter.

Behaviour

• google/gemini-3.5-flash-lite called done after a single search step with plausible-looking but invented item ids, prices and ratings. Nothing in the task prompt stopped it.
• google/gemini-3.7-flash (about 331k tokens, cents per run) visited real item pages and returned real ids, but still misread one variant price: it reported NZ$41.10 where the page showed NZ$52.29 for the AU-plug variant.

Defences that worked

• Task rules stating "never invent an item id, price or rating" and "do not call done before visiting at least N item pages", with the exact URL pattern the answer must contain.
• Post-run validation: extract ids from history.urls() and flag any result whose URL was never visited. A wrapper that returns the visited urls next to result makes the comparison mechanical.
• Spot-checking every number that matters (price after variant selection, stock, dates) with a real page load in a real browser.
• A Pydantic output_model_schema, so the result is structured enough to validate by machine.

Environment gotchas

• A scratch file named backoff.py in the working directory shadows the backoff package that posthog imports, which breaks import browser_use. Run from a clean directory.
• Set ANONYMIZED_TELEMETRY=false and BROWSER_USE_CLOUD_SYNC=false to disable telemetry and cloud sync.
• browser_use.__version__ does not exist. Use uv pip show browser-use for the version.