November 5, 2026 · RAG · ~9 min read

Multimodal RAG on 3,450 construction drawing pages: Nemotron 3.5 Embed + Rerank vs BM25, Ovis-VL and WeMM

Research test, November 2026 · run locally on NVIDIA GB10 · 101 questions with an answer

Multimodal RAG on construction documents has one hard problem: much of what an engineer needs sits on drawings with no text layer. We built a local retrieval pipeline on 3,450 pages of real tender documentation and measured it question by question against plain keyword search. The full pipeline put the right page at rank 1 for 61% of questions (BM25: 37%) and in the top 5 for 81% (BM25: 54%). This is NVIDIA's retrieval pair (Nemotron 3.5 Embed + Rerank), not the Nemotron 3.5 Super VL language model we tested in How to choose an AI model.

The result, layer by layer

This is a RAG benchmark of the retrieval step only: does the right page reach the model that writes the answer? We started from keyword search and added one layer at a time, measuring every variant on the same 101 questions that have an answer in the documents.

101 questions with an answer in the documents (106 in total). Share of questions where the right page appears at rank 1, in the top 5 and in the top 10.

The visual embedder lifts the top 5 the most (54% → 77%). The reranker lifts rank 1 the most (45% → 61%). Adding OCR text cost two points in the top 5 but bought more first-place hits, which is what a person reading one answer cares about.

Dataset: real tender documentation, drawings with no text layer

The test library is public tender documentation for two nearly identical sports halls in Poland, plus 12 official manufacturer manuals (among them an excavator, a heat pump and an air handling unit) and 300 photos from Wikimedia Commons and the BD3 dataset. It is a visual document retrieval problem as much as a text one:

• 633 PDF files, about 3,450 pages, 850 MB, about 4,300 items in total.
• Hundreds of pages with no text layer: scanned sheets and drawings. A text-only RAG over PDFs sees nothing on them.
• A1 and A0 sheets with small annotations, split into tiles so details are not lost.
• Bills of quantities and cost tables (CSV), site reports (DOCX) and 32 site voice notes, transcribed with Whisper. The voice notes (synthetic voice) and some reports were generated for the test from the project.
• 106 engineering questions with reference answers, 101 of them answerable from the documents. Some categories have only 3–10 questions.

Research on RAG over engineering drawings is only starting; ORNL's BLUEPRINT also combines keywords and vectors on drawing archives.

Embedders compared: Nemotron 3.5 Embed 8B, Ovis-VL 9B, WeMM 9B, Ovis-Omni 3B, BM25

Each multimodal embedding model indexed every page as an image together with its text, and we measured it alone, with no reranker. BM25 is the keyword baseline.

Embedder alone, no reranker, same questions. Model cards: Nemotron 3.5 Embed 8B, Nemotron 3.5 Rerank 8B.

The best embedding model for RAG on this data was not a combination. Merging several embedders did not improve results; one strong visual embedder plus OCR text, keywords and a reranker did. A second model only earned its place for search by photo, where WeMM found the document from a photo in 50% of cases and Nemotron Embed in 0%.

Hybrid search + reranker: 45% → 61% at rank 1

The pipeline runs in three steps:

1. Indexing, once per document. Each PDF page goes into the index as an image with its text. Pages with no text are read by our local OCR, large sheets are tiled, DOCX and CSV become text and voice notes become transcripts. Nemotron 3.5 Embed 8B writes one vector per item into a vector database, and a keyword index is built alongside.
2. Retrieval, on every question. Hybrid search combines vectors and keywords into about 50 candidates. A language model first splits complex questions into 2–3 separate searches. The reranker, Nemotron 3.5 Rerank 8B, then reads each candidate together with the question and reorders them.
3. Answer. A vision-capable language model (DeepSeek V4.1 Flash, running locally) reads the images and text of the best few pages and answers with source numbers, or says the information is not there.

Keywords still matter: BM25 is best at exact amounts, dates and codes. Routing tender questions about amounts and deadlines towards keyword weight took that group from 0% to 67% at rank 1. Chunking long pages put the right page among the candidates for 86% of questions.

Query format changed the hit rate from 53% to 95%

The most quotable lesson of this test: how you hand the query to an embedding model decides the score. For one of the models, three formats of the same query gave very different hit rates on a sample of questions.

Every model has to be calibrated before it is judged. Without that, a RAG evaluation measures query formatting, not model quality, and a weaker model with the right format can beat a stronger one with the wrong one.

Why not ColPali?

We did not test ColPali, so there are no ColPali numbers here. ColPali and ColQwen take a different route to visual RAG: they keep many vectors per page (one per image patch) and score them against the query with late interaction, ColBERT-style. That keeps fine detail, but a multi-vector index is many times larger than one vector per page.

We chose a single strong embedder per page, then put OCR text, keywords and a reranker on top. On 3,450 pages that reached 81% in the top 5, and the single-vector index for a million pages would take about 8 GB. A head-to-head with ColPali on the same questions is a fair next test.

Throughput on one NVIDIA GB10

Retrieval ran on one NVIDIA GB10 unit, the chip inside DGX Spark and the ASUS Ascent GX10 (see ASUS GX10 vs DGX Spark), with about 119 GB of memory. Answers ran on a second unit or a small cluster (see our DGX Spark cluster page).

Times are indicative: the GPU was shared between jobs during testing. The reranker is the first thing to speed up, with fewer candidates or a smaller model.

What still fails

• A question about the bid bond amount and the completion date in the tender specification does not reach the right page.
• Search by photo: 0% with Nemotron Embed, 50% with WeMM, and not integrated into the pipeline yet.
• Drawing revisions: 67%. The fix is metadata and filters (project, trade, revision, file type), not a better embedder.
• Some settings (keyword weight, the routing rule) were tuned on the same test questions, so the results are slightly optimistic.
• The questions were asked in Polish, over Polish documentation and English manuals. Polish questions found the right page of an English manual at rank 1 in 86% of cases; English-only question sets were not measured.
• The examples we quote elsewhere are the best cases. The numbers in this article cover all 101 answerable questions.

Fine-tuning on your own documents

These numbers come from off-the-shelf models. NVIDIA reports over 10% improvement in Recall@10 and NDCG@10 after fine-tuning an embedding model on domain data, and 26% in an Atlassian example on JIRA data. We have not measured fine-tuning on this dataset yet; with a client, we measure it on their own documents. Two more things are on the list: a lighter embedder (Nemotron 3.5 Embed 1B in NVFP4, which NVIDIA puts at about 95% of the 8B quality) and metadata filters for revisions and projects.

FAQ

Which embedding model is best for RAG on construction drawings?

In our test on 3,450 pages of tender documentation, NVIDIA Nemotron 3.5 Embed 8B put the right page in the top 5 for 77% of questions without a reranker, ahead of Ovis-VL-Embedding 9B (68%), WeMM-Embedding 9B (62%), BM25 keyword search (54%) and Ovis-Omni-Embedding 3B (37%). WeMM was the only one useful for search by photo.

Does a reranker help in multimodal RAG?

Yes, most of all at rank 1. Adding Nemotron 3.5 Rerank 8B on top of visual embedding, OCR text and keywords raised the share of questions with the right page at rank 1 from 45% to 61%. The cost is time: about 18 s per question with 50 candidates on our setup.

Did you test ColPali?

No. ColPali and ColQwen store many vectors per page and score them with late interaction; we chose one strong single-vector embedder plus OCR text, keywords and a reranker. We have no ColPali numbers on this dataset.

Can multimodal RAG run on a single NVIDIA GB10?

Retrieval can: embedding, reranking and OCR run on one GB10 unit with about 119 GB of memory, which indexed about 2.5 pages per second. Answers were written by a language model on a second unit or a small cluster.

Results cover this dataset, these questions and this configuration in November 2026. Before you choose a model for your documents, measure it on your own questions.