LLM product matching: an identity judge is not a substitution judge

Two different questions

An LLM matcher can answer two very different questions, and using the wrong one silently lowers recall:

• Identity: "are A and B the same product?" This suits reconciling a receipt line to a specific SKU.
• Substitution: "is B a reasonable substitute for A?" This suits "what would this basket cost at another store".

A judge written for identity rejects good cross-brand equivalents. An identity-framed judge rated "Baby Spinach 300g" as not matching a retailer's own-brand fresh baby spinach with confidence 1.0, and likewise a garlic bread pair, purely because the brands differed. About a third of a basket dropped to "no match", and since unmatched lines kept their original price, the other store looked falsely expensive.

Fix: a substitution-framed prompt

Verify with a prompt that is explicitly brand- and size-agnostic and gives concrete accept/reject examples:

> Is B a reasonable substitute for A, the same TYPE of grocery product, so the shopper would buy B instead? IGNORE brand and pack size. Examples: "Retailer Baby Spinach 250g" vs "Baby Spinach 300g" = yes; "Standard Milk" vs "Lite Milk" = no; "Bananas" vs "Banana Baby-Food Pouch" = no. Reply JSON {match:true/false}.

The examples teach that type matters while brand and size do not, and still reject cross-type collisions (baby food for bananas, dip for spinach, grapefruit beer for grapes). This moved one basket's match rate from 65% to 86%. A shared judge built for receipt reconciliation is too strict for substitution; a second task-specific judge avoids the problem. Pairing either judge with a call that omits sizes keeps different pack sizes from counting as a difference.

Substitution judges need a null branch and structural guardrails

Even with substitution framing, a model that must return something from a candidate list reaches for anything category-adjacent. Observed cases: a wild-caught whole crab matched to a "crab flavoured nuggets" snack (the only crab-named item, confidence 0.9), and "Extra Fancy Mixed Nuts" matched to "Mixed Cassava Vege Crisps" because "mixed" appeared in both.

Guardrails that catch this before or after the model:

• An explicit null branch in the prompt: "If no candidate is a reasonable substitute, return {stockcode: null, confidence: 0}", plus concrete accept/reject examples.
• Pre-filters before the LLM:
• Same canonical unit basis (refuse mass to volume comparisons).
• Price ratio within a plausible band (for example 0.15x to 7x the reference).
• Form must match: fresh to fresh, frozen to frozen, ambient to ambient.
• Same normalised brand, or a known private-label alias between retailers.
• A post-sweep over-use audit: GROUP BY chosen_stockcode HAVING COUNT(*) > 1 catches one target chosen for many unrelated items, which is almost always the model reaching when nothing better exists.
• Cross-region SKU sanity: the same numeric item code in independently run regional catalogues is almost always a collision between separate ID spaces. Confirm with name overlap and price ratio.

Raising the confidence threshold alone (0.7 to 0.85) helped but did not fix the "only one candidate, so it must be it" pattern: the model still returned 0.90 on a bad match because from its narrow view it was the closest option. Structural pre-filters and the null branch are what catch it. Also worth watching: a judge returning verdict "error" being coerced into a false "no match".