What DocPro search does when the embedding model is down

The first decision we made about semantic search in DocPro was not which model to use. It was what search should do when the model is unreachable.

A document search box has a hard constraint most retrieval demos ignore: it can never return an error page. A lawyer typing a matter name into DocPro at 9pm does not care that an embedding sidecar failed a health check. They care that the box returns rows. So before anything else, we decided that a degraded search is acceptable and a broken search is not, and we built outward from there.

The model is ours, and it runs beside the API, not inside it

DocPro's search is backed by an embedding model we host: a LoRA adapter over intfloat/multilingual-e5-small, producing 384-dimensional, L2-normalized vectors. We serve it from a small sidecar process that exposes an OpenAI-compatible /v1/embeddings endpoint, bound to localhost. The application never imports torch. It makes an HTTP call to the sidecar and gets vectors back.

That split is deliberate. The embedding runtime has a heavy, GPU-adjacent dependency tree with its own upgrade cadence; the API is a lean web service we redeploy several times a week. Keeping them in separate processes with separate virtual environments means a model upgrade never drags the API's dependencies with it, and an API redeploy never restarts the model. The sidecar keeps serving straight across a backend deploy.

Storage is Postgres with pgvector. Each document's extracted text is chunked to at most 256 tokens using the model's own tokenizer (not a whitespace approximation, the real one), embedded chunk by chunk, and written alongside the job. Queries embed the same way, we pull nearest neighbours with an HNSW index, and only then filter by tenant_id. Tenant isolation is a WHERE clause applied after the vector search returns candidates.

The fallback

When the sidecar is unhealthy, or a single query embedding fails, search does not raise. It falls back to an ILIKE scan over the stored text preview. Worse results, same contract: the caller gets rows, never a 500.

This shows up in two places. On read, a failed query-embed silently drops to keyword matching for that request. On write, embedding is best-effort and never fails the ingest job. If the sidecar is down when a document finishes processing, the job completes anyway and its row keeps embedded_at = NULL. That null is not a lost document. It is a work item. When the sidecar comes back, a backfill pass finds every embedded_at IS NULL row and embeds it, so the index heals itself without anyone re-uploading anything.

The backfill has one guardrail worth stating plainly: it refuses to run unless the sidecar reports healthy first. We would rather index nothing than write garbage vectors from a half-initialized model and have to tell them apart later.

Reproducibility is a version-pin problem

The subtle failure mode with a hosted embedding model is not that it breaks. It is that it quietly changes. If the base checkpoint moves underneath the adapter, the same document embeds to a slightly different vector, and a stored index built last month no longer sits in the same space as today's queries. Nothing errors. Results just get worse for reasons no log line explains.

So we pin the base model to an exact revision, and the adapter records the base checkpoint and revision it was trained against. The sidecar serves from an offline cache at that pinned revision. Rebuild the box from scratch and the vectors come out identical.

We learned one piece of this the unglamorous way. The served model directory needed the base tokenizer copied into it explicitly, or the sidecar would resolve a subtly different tokenization at load time. That is now codified in the provisioning script. It is the kind of bug that produces no error and a small, permanent accuracy tax until you find it.

What we have not measured yet

Here is the honest gap. Our published multilingual embedding evaluations are cross-lingual retrieval over Wikipedia pairs across ten Indian languages. That is a real benchmark with reproducible code, but it is not the same distribution as a law firm's contracts and filings. We do not yet have a public DocPro-domain retrieval benchmark, and until we do, we are not going to quote a single recall number and imply it describes production document search.

We also build a legal-domain English embedding adapter, embed-legal-en, published under Apache-2.0. It is not wired into DocPro today; it runs as a live base-vs-adapter comparison in our public playground, where you can paste a legal query and watch the domain-adapted vectors reorder the results against the generic base. Promoting a domain adapter like that from a demo into DocPro's retrieval path is a specific piece of work, not a thing we can claim by association. When it ships into the product, we will say so here, with the before-and-after on real documents.