The retrieval number we took back before we published it

We built a cross-lingual retriever for EU law. Give it a passage of EU law in one language and it finds the matching provision in another, across German, English, Spanish, French and Italian. On our held-out set it lifted recall@1 from 0.2919 to 0.6480, a relative gain of 122 percent, with confidence intervals that do not overlap.

That is the number we are publishing. It is not the number we first measured.

The first time we trained this adapter it scored +126.3%. We are not hiding that number, and we will do on the page the thing you are about to do with it. Put +126.3% next to +122.0% and subtract: 4.3 points. That 4.3 is not what the leak cost us. The +126.3% was measured on a split that held nothing out, so it never measured cross-lingual generalisation at all: it is a number for a different quantity, not a larger version of this one. The two runs were not trained on the same amount of data either. Subtracting two numbers that measured different things on different training sets does not produce a quantity that exists. We retracted +126.3% because the split behind it leaked, not because it was high by a knowable margin. Both figures were measured on the same 5,509 pairs against the same 0.2919 baseline; what changed was the training set.

Here is the leak in plain terms. A cross-lingual corpus stores each alignment in both directions: Article N in German paired to Article N in Italian is also present as the Italian paired to the German, the same two passages with their roles swapped. Our split held out one direction and left the other in training. To a model that treats both directions the same way, that is the answer sitting in the study guide.

So we fixed the split to hold out whole units instead of single directions, and re-ran on data that does not leak. Two things are true about the new number and belong together: it is smaller, and it is the first one that measures generalisation at all. The clean run also trains on a little over half as much data, because holding out whole units costs volume. We are publishing the one we can stand behind.

We caught this one before it shipped. In the same week, five models we had already published turned out to carry numbers from leaked splits. Three we could correct: embed-gov-indic to +37.1%, embed-legal-en to +59.7%, embed-statute-en to +28.1%. Two we could not. Both of our reranker numbers were withdrawn outright, because their splits excluded nothing at all: there was no smaller true number to fall back to. One of them, rerank-statute-en, has since been retrained and re-measured on a clean, document-isolated split, and now has a valid effectiveness number for the first time: +47.2% recall@1 over the bi-encoder, paired 95% CI [+0.0207, +0.0482]. That is not a corrected version of the withdrawn figure. The withdrawn figure measured nothing, so there was nothing to correct; it is a different model, on a different split, reported from zero. That +47.2% is a single training draw: the run seeds its data split but not its training, so it is one measurement of one checkpoint, not a property of the recipe. The second, rerank-gov-indic, has since been retrained and re-measured the same way, from zero on a clean split. Read that checkpoint as an absolute before a percentage: recall@1 goes from 0.0626 to 0.1291, a measured gain of 0.0665 with a paired 95% CI of [+0.0532, +0.0807]. It is a weak first stage, not a strong retriever, and most queries whose answer is already retrievable still miss the top rank; of the ones that are retrievable at all, the reranker puts 13.9 percent in first place. As a relative figure that draw is +106.4%, large only because the number it doubles is so small. One caution belongs on it that does not belong on the headline above. When we re-ran that identical arm from zero on byte-identical data, a second draw came back the other way: a significant regression of -0.0430 (-68.8%), paired 95% CI [-0.0546, -0.0311], on the same split with the same mining ledger. The spread within that one arm is 95.7% of the spread between our best and worst arms, so this evaluation cannot separate an arm effect from a draw effect. The +106.4% is not withdrawn and neither draw replaces the other; what we cannot yet say is that the recipe, rather than the draw, produced it. Like the statute reranker, it is reported from zero, not a recovery of the figure we withdrew. This post's number never shipped inflated, because the check ran before the post did. That is the only difference worth claiming: not that we never make the mistake, but that this time the order was right. All five cards are public and each carries its own correction: embed-gov-indic, embed-legal-en, embed-statute-en, rerank-statute-en, rerank-gov-indic.

Two things about the surviving number, stated as what they are and no more:

• Because the two sides of every pair are in different languages, almost every pair in this evaluation already has near-zero word overlap, 5,217 of 5,509. So the headline is already the low-overlap number; there is no easy majority of vocabulary-sharing pairs propping it up. A system leaning on word overlap has very little to work with here, and the model clears it anyway.
• The gain is even across languages: 0.653 in German, 0.648 in English, 0.640 in Spanish, 0.648 in French, 0.651 in Italian. That flatness means one specific thing: no single language is being carried by the average of the others. We claim nothing beyond that about the shape of the space.

Two conditions on that headline number, because a post about measurement owes them. It is a single training draw, and the run is only partly seeded. The shuffle and the dropout draw from a configured seed, but the adapter's own down-projection is initialised before that seed is set, so a re-run would start from a different adapter and we cannot promise it lands on the same number. That is not a claim it would move. It is a LoRA adapter over a pretrained model, not the freshly initialised ranking head where we have watched two draws of the same arm disagree, and we have not measured this adapter's run-to-run spread. We are stating the draw count, which is one. What we cannot say is that a second draw would be a re-run of this one. The corpus it trained on is still on disk, but the exact training tree is not recoverable as a commit: the working copy of the training module on the box was written after the adapter it is supposed to have produced, so the file on disk is provably not the file that ran. We can retrain on the same data with the package we publish; we cannot reconstruct this run, and the model card says so. We have not repeated it. Separately, the evaluation scored passages under a 256-token cap, and we have not measured how often a relevant passage was truncated at it; +122.0% is the number under that setting.

One boundary, stated plainly, because stating it is the point. This result is in-distribution: same corpus, same task, EU law measured on EU law. It is not a claim about another body of law or another domain, and we are not implying one. When we have a number for something else, we will measure that thing and publish that number.

A note on what this is: a research artefact, a retriever we trained and measured on EU law, not the general multilingual model that runs inside our product. It is a LoRA adapter over intfloat/multilingual-e5-small, published at https://huggingface.co/quanfire-ai/embed-eulaw-multi, and the source is EUR-Lex, the EU's own open legal database. That is the same argument our earlier post on training data made, turned toward measurement instead of licensing: clean data, honestly measured. A benchmark is only worth what the split behind it is worth. This one now is.