Why Embeddings Beat Filters (Then Don't)

Semantic search found recipes keyword search never could. Then it started recommending pasta for every soup query.

FoodNet has ~2,000 recipes. Users search with natural language: "quick high-protein dinner", "something with the chicken I have", "comfort food for a cold night." Keyword search can't handle this. But pure embedding search has its own problems.

Here's how we built a hybrid search that works.

The Hybrid Architecture

Every search query produces scores from four independent signals, weighted and summed:

Signal	Weight	What it captures
Semantic (embedding cosine)	6.0	Conceptual similarity
Full-text search (tsvector)	3.0	Keyword presence
Exact title match	3.0	Direct hits
Trigram similarity	2.0	Fuzzy spelling tolerance

Plus two quality modifiers:

• Source tier (0.2) — recipe quality ranking
• AI score (0.5) — multi-persona appeal/practicality composite

Each signal retrieves its own candidate pool (200 semantic, 100 FTS, 50 trigram), then scores are normalized within each pool before the weighted sum. This prevents any single signal from dominating through scale differences.

Where Embeddings Win

Semantic search handles queries that keyword search cannot:

• "something light for summer" → finds salads, ceviche, gazpacho (no keyword overlap)
• "kid-friendly weeknight" → finds mac and cheese, chicken nuggets, pasta (intent, not ingredients)
• "meal prep for the week" → finds batch-cookable stews, casseroles, grain bowls

These work because OpenAI's text-embedding-3-small (1536 dimensions) captures meaning, not just tokens.

Where Embeddings Break

The problem: most recipes share the same base ingredients. Salt, oil, butter, flour, eggs, garlic, onion. When you embed a recipe's full ingredient list, these pantry staples dominate the vector. The result: every recipe looks similar.

We discovered this when "chicken soup" queries started returning pasta recipes — because both use salt, oil, garlic, and onion.

The Fix: Ingredient Noise Filtering

We maintain two lists:

HARD_DROP — 25 ingredients always excluded from embeddings:

salt (table, sea, kosher), black pepper, white pepper,
water (cold, warm, hot, boiling, ice),
cooking spray, nonstick spray

BORDERLINE — 95 ingredients demoted to end of list unless title-reinforced:

oils (olive, vegetable, canola, sesame, coconut),
fats (butter, margarine),
flour, sugar, eggs,
garlic, onions (all colors)

Title reinforcement: if any word in an ingredient appears in the recipe title, it stays in its original position. "Garlic Butter Shrimp" keeps garlic and butter front and center. "Chicken Parmesan" demotes the olive oil and flour.

Author Name Pollution

Another embedding failure mode: "Mary's Famous Chicken Soup" embedded closer to other "Mary's" recipes than to chicken soups. The embedding space was clustering by author name, not by food.

Fix: strip possessives from titles before embedding. "Mary's Famous Chicken Soup" → "Chicken Soup".

The Query Parser

Free-text queries need decomposition before they can drive hybrid search. Our parser runs 7 phases:

1. Normalize (lowercase, collapse whitespace)
2. Extract time constraints via regex ("under 30 minutes" → max_time: 30)
3. Multi-word phrase matching (longest-first) for cuisine, style, dietary
4. Single-word matching for difficulty
5. Strip filler words ("great", "best", "amazing", "classic", "homemade")
6. Remaining tokens → FTS query (& joined)
7. Preserve raw query unchanged for embedding

The vocabulary: 32 cuisine terms, 14 meal types, 17 style terms, 9 dietary terms, 7 difficulty terms, 200+ first names (to strip "Mary's" from FTS queries too).

Longest-first matching prevents false positives: "side dish" matches as a meal type before "side" can match anything else.

AI Score Integration

AI recipe scores (from our multi-persona scoring pipeline) feed into search ranking with centered normalization:

normalized = (avg_composite - 5.0) / 4.5

This means:

• Unscored recipes get 0.0 — no boost, no penalty
• Average recipes (~5.0) also get ~0.0
• Great recipes (8+) get positive boost
• Poor recipes (2-3) get negative penalty

The human shutoff at 5 interactions means: once a user has enough search/save/cook history, AI scores fade out and their own behavior drives ranking.

What We Learned

Embeddings are powerful for capturing intent. But they need guardrails:

1. Filter noise before embedding — pantry staples collapse the vector space
2. Don't embed metadata — author names, marketing copy, and brand names pollute similarity
3. Hybrid > pure semantic — keywords still matter for exact matches and spelling
4. Normalize across signals — raw cosine similarity and raw BM25 scores live on different scales

The search weights (6.0 / 3.0 / 3.0 / 2.0) were tuned empirically. Semantic leads, but it doesn't dominate.

---

Next: AI Scoring Systems Drift — why our first scoring run was useless and how comparative batching fixed it.