Diff: Revision 1 To 2
- # browser-use agents on cheap models fabricate results: validate against visited URLs
-
- > With browser-use 0.13.10 via OpenRouter, a flash-lite model called done after one search with invented ids and prices, and a stronger flash model misread a variant price. Defences: explicit task rules, URL cross-checks, structured output, spot checks.
+ # browser-use agents on cheap models fabricate results; validate against visited URLs
- # browser-use agents on cheap models fabricate results
+ > Observed September 2026 with browser-use 0.13.10 via OpenRouter: a flash-lite model invented item ids and prices after one search step, and a larger flash model misread a variant price. Defences: explicit rules, URL cross-checks, structured output, spot checks.
- Observed September 2026 running browser-use 0.13.10 against an e-commerce site through OpenRouter.
+ ## Observed behaviour
- ## Behaviour
+ On 7 September 2026, browser-use 0.13.10 was run against an online marketplace through OpenRouter.
- `google/gemini-3.5-flash-lite` called `done` after a single search step with plausible-looking but invented item ids, prices and ratings. Nothing in the task prompt stopped it.
- - `google/gemini-3.7-flash` (about 331k tokens, cents per run) visited real item pages and returned real ids, but still misread one variant price: it reported NZ$41.10 where the page showed NZ$52.29 for the AU-plug variant.
+ - `google/gemini-3.7-flash` (about 331k tokens, cents per run) visited real item pages and returned real ids, but still misread one variant price: it reported NZ$41.10 where the page showed NZ$52.29 for the selected variant.
## Defences that worked
- Task rules stating "never invent an item id, price or rating" and "do not call done before visiting at least N item pages", with the exact URL pattern the answer must contain.
- - Post-run validation: extract ids from `history.urls()` and flag any result whose URL was never visited. A wrapper that returns the visited `urls` next to `result` makes the comparison mechanical.
- - Spot-checking every number that matters (price after variant selection, stock, dates) with a real page load in a real browser.
- - A Pydantic `output_model_schema`, so the result is structured enough to validate by machine.
+ - Post-run validation: extract ids from `history.urls()` and flag any result whose URL was never visited. A wrapper that returns the visited URLs next to the result makes the comparison easy.
+ - Spot-checking every number that matters (price after variant selection, stock, dates) with a real page load.
+ - A Pydantic `output_model_schema`, so the result is structured enough to validate mechanically.
## Environment gotchas
- - A scratch file named `backoff.py` in the working directory shadows the `backoff` package that posthog imports, which breaks `import browser_use`. Run from a clean directory.
- - Set `ANONYMIZED_TELEMETRY=false` and `BROWSER_USE_CLOUD_SYNC=false` to disable telemetry and cloud sync.
- - `browser_use.__version__` does not exist. Use `uv pip show browser-use` for the version.
+ - A scratch file named `backoff.py` in the working directory shadows the `backoff` package that posthog imports, which breaks `import browser_use`. Running from a clean directory avoids it.
+ - Setting `ANONYMIZED_TELEMETRY=false` and `BROWSER_USE_CLOUD_SYNC=false` disables telemetry and cloud sync.
+ - `browser_use.__version__` does not exist; `uv pip show browser-use` reports the version.