Case studies

Engineers search docs with both intent ("why does my proxy return 502") and exact tokens (flags, error codes, file names). Dense models handle the first, sparse models the second, and neither handles both.

Two indexes: 320-token chunks stored twice in Qdrant: 1024-d BGE-large vectors in HNSW and SPLADE sparse vectors in an inverted index

Fusion: Weighted RRF with 1-based ranks, k = 10 selected on the dev split, computed client-side after probing Qdrant’s built-in RRF

Dynamic weights: Queries with identifiers, paths or error codes lean towards SPLADE; natural-language questions lean dense

Answers: A router picks the local index or SearXNG web search; Gemma 4 12B answers only from numbered passages and cites them

0.371 — nDCG@10 on BEIR CQADupStack test (BM25: 0.293)

0.761 — Recall@100, dev-tuned weighted RRF

50ms — p50 retrieval: both encoders, Qdrant, fusion

0.2ms — p50 RRF fusion

↖ fuse ranks, not scores

"Model B looks better" is usually decided on a handful of examples. Teams need to know whether a change is a real improvement, what it costs, and when to fail CI.

Suites: YAML prompt templates, cases and weighted assertions; every edit saves an immutable version

Comparisons: Paired-bootstrap 95% CI for the score and McNemar’s test for pass/fail; below 10 paired cases it refuses to declare a winner

Cost: Provider-reported token usage times your own prices; unpriced models are shown as unpriced, not $0

CI: The llmwatch CLI exits non-zero only on a significant regression against a baseline run

10 — evaluators, from regex to LLM-as-judge

10,000 — paired-bootstrap resamples per comparison

1 cmd — docker compose up: API, UI and SQLite on one port

↖ no winner without significance

Coordinating multi-step workflows across crashing workers and replicas usually means either lost work or work that runs twice.

Event log: Every state change is appended in the same transaction as the tables it updates; replaying the log reproduces them exactly

Fenced leader: A Postgres lease with fencing tokens; a trigger rejects writes from a stale leader

Claims: SELECT … FOR UPDATE SKIP LOCKED under heartbeated leases; results accepted only from the current attempt

Sagas: If a step fails for good, completed steps are compensated one at a time, last first

~450 — claimed tasks / s on a laptop

~10s — leader failover (lease TTL)

204 — labeled failures in the decision eval

↖ the log is the source of truth

Search and RAG over recorded lectures needs more than a transcript: what was on screen, and the exact moment an answer came from.

Decode once: FFmpeg resamples audio to 16 kHz mono and samples one frame per second into overlapping windows

Read the screen: Tesseract reads up to 6 selected frames per window; an ONNX image model embeds one patch per text line

Transcribe: whisper.cpp word timings shifted onto the media timeline

Package: A versioned, bounds-checked .vec file; load-to-llm renders it as Markdown, JSON or text

45s — windows with 5s overlap

ms — timestamps on words, frames and embeddings

0 — Python dependencies at run time

↖ keep everything, point back to the moment