← All posts
Research & Engineering

Don't take our word for it: read the tokens yourself

Don't take our word for it: read the tokens yourself

Most embedding models ask you to trust a leaderboard. We'd rather you check.

Today Quanfire's multilingual embeddings are public on Hugging Face, and there's a live playground where you can watch the model read your text, in any of 100+ languages, computed entirely in your browser, nothing sent to a server. Type a Hindi sentence, a Tamil one, a code-mixed one, and see exactly how it's tokenized and how meaning lines up across languages. playground.quanfire.ai

Two models are live:

multilingual-embedding is our flagship general-purpose retriever: one meaning, searched across many languages. It's Indic-first (Hindi, Bengali, Gujarati, Kannada, Malayalam, Marathi, Sanskrit, Tamil, Telugu, Urdu) and holds up globally across 15 more (English, French, German, Spanish, Arabic, Chinese, Japanese, and others). On FLORES-200, all-pairs cross-lingual recall is 0.976; French retrieval 0.990; non-Hindi Indic cross-lingual recall@1 0.899. Openly licensed under CC BY-SA 4.0.

embed-legal-en is a specialist that shows what domain depth buys you. On Indian Supreme Court judgment retrieval it lifts recall@1 by +76% over its base, and on the hardest low-lexical-overlap queries recall@1 more than doubles (0.156 to 0.325). We card it honestly: this is a judgment-domain result, not a "legal English" one. On statutory text the adapter shows flat transfer, and we say so plainly, because that's exactly what a buyer needs to know. Apache-2.0.

The part we care about most: where the data came from

Both models are trained only on openly-licensed, commercially-clean data, documented source by source: Wikipedia langlink pairs, BPCC-mined bitext (CC0), itihasa, Tatoeba for the flagship; public-domain Supreme Court judgment text (with editorial headnotes excised) for the legal model. No scraped corpora of unknown origin. When we say the provenance is clean, you can read the list.

We don't just claim it. We show the work. Here's exactly how we earned the licence on the legal model, one cut at a time: The judgment is public domain. The headnote is not.

That discipline isn't a constraint we tolerate. It's the product. Clean, documented provenance is what makes these models safe to build a business on, and it's why we can show our work instead of asking for trust.

And the quality isn't a mystery either: built from scratch, finetuning on the right clean data more than doubled retrieval on our held-out Hindi benchmark (0.105 to 0.259, non-overlapping confidence intervals). Quality here is a dial you turn with compute and clean data, not a black box.

Pull the models: huggingface.co/quanfire-ai · Verify them yourself: playground.quanfire.ai/embeddings

DocPro

Want this working on your own documents? DocPro is built by the same team, on the same provenance-first, Indic-first, verify-yourself discipline you just saw in these open models.

Share

Keep reading