Docling vs ColPali: multimodal RAG in production

Quicksort organized a meetup in Paris, where I presented multimodal RAG in production. Docling is a popular document-ingestion solution with its own merits and limitations. The same applies to ColPali, which proposes a different kind of solution entirely. In this article, I'll show the results of running both approaches, and how to choose what works better for you.

Introduction

Multimodal retrieval is the ability to retrieve visual information from documents. In many cases you don't actually need a multimodal solution: the images may not be informative, or the information they carry may already be redundant with the surrounding text.

A case where you do need the visual information is financial documents with charts, organigrams, illustrations, etc. that contain information not present anywhere else in the document.

Docling (top) vs ColPali (bottom).

Docling, on one hand, converts everything into a textual format (tables to markdown, images to descriptions, charts to tables, etc.) so we can process it afterward in a unified way, the same as classical RAG.

On the other hand, ColPali treats each page of a document as a single image, whether it's composed of text, tables, images, or all three.

We'll explore each approach in detail, and what this choice means for a production system, but before that, we need to take a look at our data.

The corpus

Imagine you want to build an AI assistant for financial documents, so risk managers or investors can use it to pull out useful information.

Fortunately, public companies are required to share their financial documents. Among those, we find Universal Registration Documents (URDs), which state a company's financial situation for a given year. We also find investor presentations, and so on.

We selected a subset of TotalEnergies' public documents and built a corpus of thousands of pages, filled with images and tables and laid out in a complex structure.

TotalEnergies' public filings, by the numbers.

Here's a look inside four of them, dense financial tables, KPI scorecards, case-study photography, and regional maps:

Four real pages, four different layouts.

For evaluation, I prepared a dataset of 76 questions against this corpus, each tied to a specific gold page and tagged by modality:

{
  "id": "q001",
  "question": "What was TotalEnergies' LNG sales volume in 2022, in million tonnes?",
  "modality": ["chart"],
  "gold_pages": [{ "page_no": 8 }],
  "notes": "Answer: 48.1 Mt, from the 'LNG sales volumes' bar chart on p.8."
}

We ran both Docling and ColPali as the ingestion solution and pushed the pipeline end to end through the same other components:

• BM25 keyword search, implemented in PostgreSQL.
• voyage-finance-2, Voyage AI's embedding model optimized for finance (Docling only).
• rerank-2.5, Voyage AI's reranker model.
• Qdrant for vector indexing.
• LangGraph to orchestrate the agent.

Docling

Docling is an open-source ingestion solution developed by IBM. It runs a pipeline of multiple models to convert a complex document into a textual format:

• OCR.
• Layout detection.
• TableFormer (converts tables to structured text).
• Image classification.
• Image description.
• Chart extraction.

Docling pipeline models.

Docling starts by running OCR on each page (if activated), but we don't since these PDFs already carry a text layer. Then layout detection finds the page's elements; on tables we run TableFormer; on images we run classification, description and chart extraction if activated.

Docling state diagram.

As you'd expect, running Docling is slow, digesting the corpus can take hours. Time per page is far from uniform: it depends entirely on which elements get detected.

Docling processing time per page.

For deploying Docling, there's a ready-to-use Docker image, docling-serve-cu128, for serving it on CUDA 12.8. I used an A40 GPU from Runpod at $0.5/hour. The GPU has 48GB of VRAM, which I expected would be plenty to serve Docling.

However, the base image serves its models through transformers. The pipeline was very slow, so I decided to serve the chart-extraction model (4B parameters) separately using vLLM and connect it to the main Docling server.

Ingestion runs as a background task. We process each document in batches of 50 pages, so a failure only costs the batch that broke, not the whole document: we can just retry that one batch on its own.

Once the pipeline runs, a complex, visually rich document becomes a structured markdown file, ready for chunking.

A page input to a markdown output.

As you can see, tables and charts get correctly converted to markdown, where they can easily be embedded and retrieved.

For chunking, Docling offers a HybridChunker that applies topological chunking, but I preferred unstructured's chunk_by_title function, which I tweaked to split a chunk on a maximum token count and a maximum page count, so a chunk never exceeds two pages.

The reason: a single section can span multiple pages with several images and very little text. If we only counted tokens, a chunk could span many pages and many images, which would make it much harder to represent correctly in a single embedding.

At the end of this, we have a list of chunks stored in PostgreSQL with the following shape:

{"chunk_id": str,
 "text": str,
 "page_start": int,
 "page_end": int,
 "section": str,
 "has_table": bool,
 "images": [{page, id, bbox: {x1,y1,x2,y2}}],
}

To sum it up, here are the pros and cons of using Docling:

ColPali

I won't go into ColPali's inner workings here, since I already wrote a detailed article about it: RAG with ColPali: theory, implementation, and production tips.

I'd encourage you to go through that article, but in short: ColPali treats each page as an image, using a vision-language embedding model. Each page's output isn't a single vector but many, what's known as late interaction.

This matters because ColPali replaces everything Docling does, plus the embedding step, with a single pass that generates the vectors. That means there's no chunking in ColPali: each page is its own chunk, and multiple embedding vectors represent its full content.

ColPali workflow: no layout model, no OCR, no table parser, no chunking.

We're actually using the ColQwen3-4B model, served with vLLM on another A40 GPU.

The first benefit of using ColQwen is ingestion speed: a single model, end to end. ColQwen is 5.3x faster than Docling on our full corpus (15 documents, 3,232 pages).

Ingestion time of Docling vs ColPali on all 15 documents.

However, if we want the same RAG pipeline as Docling, we need to decide how to implement hybrid search with BM25, RRF, and reranking.

For ColQwen, we don't need to extract any text from the page, but if we want to use BM25, we can just pull the page's text (using pymupdf in our case, or any other PDF library) and run BM25 over it, then fuse with RRF as usual.

Open question: what do we do if the documents are scanned? If we actually want to apply BM25 and reranking, OCR stops being optional. But if ColQwen retrieval alone is already good enough, skipping OCR entirely saves an enormous amount of processing time.

For reranking, things get more delicate. First: do we even need reranking with ColQwen? If you've read my ColPali article, you'll recall that late interaction already sits as a middle step between embedding and reranking, but nothing stops us from adding a reranker on top to improve retrieval quality further.

The main problem is that the reranker only accepts text as input. We can rerank on the page's extracted text, but then the images are left out of the scoring entirely; a chunk with an important image can score lower, or get dropped, if the surrounding text isn't very relevant.

The more serious problem is what to do with image-only chunks. Imagine a page that's just a picture: no text at all, and the reranker has nothing to work with.

As a workaround, we can rerank only the pages that contain text, and always keep the retrieved image-only pages regardless of their rerank score. In the evaluation below I kept it simple and reranked every retrieved page on its text layer.

Note that pymupdf's get_text method also picks up text that's visually part of an image, as long as the page isn't a scanned image. So a chunk with literally no text at all is rare, but irrelevant surrounding text is still a problem for reranking.

ColQwen solves multiple problems, but it introduces complications of its own in the rest of the RAG pipeline. To sum it up, here are the pros and cons of using ColPali:

Another problem is about infrastructure. ColQwen is faster and cheaper to run than Docling (a single GPU is enough), but it carries a hidden cost at query time: it's an embedding model, and there's no managed, pay-as-you-go service for it the way there is for text-embedding models. If we want to use it at query time, the server has to stay up around the clock ($0.5/hour). That's not the case for Docling, which is never invoked at query time at all. A serverless, pay-as-you-go ColQwen API would remove this problem entirely.

The final unanswered question is at generation time, after retrieval: how will the LLM actually consume the retrieved pages? Do we send them as images? Do we run some lightweight ingestion on them first?

The hybrid approach

As mentioned earlier, we still need to decide how the LLM consumes the pages that ColQwen retrieves. One option is to apply Docling to the retrieved top-k pages.

Docling's role here is narrower: it only prepares the data for the LLM. We run layout detection to identify the elements on each page, then decide what to do with each one:

• Tables: convert them to markdown with Docling. That's what I did: table structure stays on in the light pass.
• Images: no need for image description or chart extraction — the LLM can handle those directly, so each picture is cropped out of the page and sent alongside the markdown. Image classification drops useless pictures like logos, but only when they're also small: these reports draw whole charts out of icons.

The hybrid approach: ColPali for retrieval, Docling at query time.

This stays fast, since we skip the heavy models (no OCR, image description or chart extraction) and only run it on the top-k pages: 1.5–3.5 s per page in my spot checks. We could even cache the enriched version in the database, so a page that's already been processed never goes through it twice.

The downside is infrastructure: both ColQwen and Docling need to be up and running, and you pay for both at query time.

Evaluation

It's the moment of truth: which approach is better? We'll evaluate both on a dataset of 76 questions over the 15-document corpus, where the answer lives in a chart (30 questions), a table (28), text (9), or a mix of them (9).

Setup

• Two isolated pipelines. Each pipeline ingests the same 15 PDFs (3,232 pages) into its own Postgres database and Qdrant collection: Docling as 10,286 chunks embedded with voyage-finance-2, ColPali as 3,232 pages embedded with ColQwen3-4B (128-dimension multivectors).
• Dense-only retrieval: each pipeline's vector search alone (cosine similarity for Docling, MaxSim for ColPali). No BM25, no fusion, no reranking.
• Hybrid retrieval: vector search plus BM25 (Postgres full-text search over the chunk text for Docling, and over each page's text layer for ColPali), merged with weighted RRF, then reranked with Voyage rerank-2.5.
• Equal page budget: a Docling chunk covers about a third of a page on average (10,286 chunks for 3,232 pages), so comparing Docling's top 5 chunks with ColPali's top 5 pages would short-change Docling. Docling retrieves its top 15 chunks instead, and both pipelines are scored on their first 5 distinct pages. For generation, the LLM gets every Docling chunk that falls on those 5 pages.
• Retrieval metrics: the first 5 distinct pages each pipeline retrieves, with a chunk counting for every page it spans. A question is a hit if its gold page is among them. We report Recall@5, Precision@5, MRR@5, nDCG@5 and end-to-end latency.
• Generation: GPT-4o (temperature 0) answers from the top 5 pages of hybrid retrieval, and a GPT-4o judge grades each answer against a human-written reference: correct = 1, partial = 0.5, incorrect = 0.

Retrieval is deterministic: re-running all four retrieval experiments reproduced every per-question score exactly. One caveat applies to every recall number below. Each question has a single labeled gold page, but these reports repeat their key figures: the 2022 LNG sales volume (48.1 Mt) appears in a chart on page 8, a table on page 55 and a sentence on page 13. Retrieving page 55 counts as a miss, so recall understates how often the answer actually came back.

Retrieval results

ColPali wins every quality metric, with or without hybrid search. However, the aggregate hides where that gap comes from. By content type:

The gap is mostly charts. ColPali retrieves the gold chart page for 63% of chart questions in both setups; Docling manages 23% dense-only and 27% hybrid. Hybrid search barely moves Docling there, because no reranker can recover a number the parser never extracted. Everywhere else hybrid search does most of the catching up: Docling ties ColPali on text (0.89 each) and gets close on tables (0.71 against 0.75), while ColPali keeps a clear lead on mixed content.

Generation results

Retrieval recall isn't what users see; the generated answer is. We compare three setups, each using its own pipeline's hybrid retrieval:

• Docling: the Docling chunks on its top 5 pages, as text.
• ColPali (page images): the top 5 pages, rendered and sent to the LLM as images, with no parsing at all.
• ColPali + Docling (the hybrid approach): the same top 5 pages, run through a light Docling pass at query time (layout detection, table structure and picture classification; no OCR, picture description or chart extraction). Each page reaches the LLM as markdown, tables included, plus a crop of every picture that isn't decoration.

The hybrid approach produces the best answers: 0.85 mean score and 79% fully correct, against 0.80 and 71% for page images and 0.71 and 61% for Docling. The two ColPali setups answer from exactly the same retrieved pages for every question, so the difference between them comes only from how the page reaches the LLM: markdown tables plus targeted picture crops beat whole-page images.

Docling's lower score is mostly a retrieval problem: it found a gold page for 42 of 76 questions against 57 for ColPali. It still trails when it did find the gold page (0.86 against 0.90 for the hybrid approach). In exchange it is the cheapest to run: images are expensive tokens, so both ColPali setups cost nearly twice as much per question in LLM calls.

The hybrid approach beats page images on every content type, most on charts (0.85 against 0.78), where a cropped chart is easier to read than a full page, and on tables (0.89 against 0.86), where markdown spares the LLM from reading numbers off pixels. Docling keeps the lead on text questions (0.83), where clean text chunks are exactly what the LLM needs; with only 9 text questions, that gap is a handful of answers.

Missing the gold page lowers the score without zeroing it. Every answer judged correct after a miss cites a retrieved page that restates the same figure, which is the gold-label caveat from the setup, not the LLM answering from memory.

A note on fairness: Docling's numbers could likely improve with more tuning: a different choice of models (picture description, chart extraction), a different chunking strategy, a different top-k. But that tuning is part of the cost. ColPali reached these results with a single model, no chunking and almost nothing to configure, and that simplicity is part of the comparison, not just accuracy.

Failures

Aggregate numbers say how often something fails, not why. Here are three questions from the 2022 report.

The first is a ranking failure. Page 100: a 37-row table of pipeline interests worldwide. The question asks for the stake in the Dolphin pipeline, Qatar to the UAE, which is the table's very last row.

The extraction is perfect: the whole table in one chunk, every row intact.

| Middle East and North Africa |  |  |  |  |  |  |
| United Arab Emirates |  |  |  |  |  |  |
| Dolphin | North Field (Qatar) | Taweelah-Fujairah-Al Ain (United Arab Emirates) | 24.50 |  |  | X |

Dense-only, Docling still misses it (recall 0.0): the embedding of a 37-row numeric table doesn't look much like a question about one pipeline. With hybrid search, BM25 matches "Dolphin" and the reranker reads the row, and the same chunk makes the top 5, in second place (recall 1.0). ColPali finds the page in both setups.

The second is an extraction failure, then a reading failure. Page 15: a stacked bar chart of projected 2030 energy sales. The question asks for the combined share of electricity and low-carbon molecules, bracketed on the chart as 20%.

Docling's table-structure model mistook the chart's gridlines for a data table. Verbatim, from the chunk store:

| 2025 | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 2030 | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 2022 | | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
...

"20%" appears nowhere in the text, and Docling misses the page in both setups. ColPali retrieves it in both. But all three generation setups still get the answer wrong, in the same way: GPT-4o reads the bracket as 20% each for electricity and low-carbon molecules, and answers 40%. Retrieval solved this question; reading the chart didn't.

The third is a classifier that threw the chart away. Page 8: a page of infographic charts, each drawn from icons. The question asks for the 2022 LNG sales volume, 48.1 Mt, shown in the flame-icon chart.

Page 8 of the 2022 report: every chart on the page is classified as an icon.

Docling's picture classifier labels the LNG chart an icon, so the chart never reached the chunk store: none of page 8's chunks contain "48.1" or "LNG sales". The question was still answered correctly by all three setups, because 48.1 Mt is restated on pages 13 and 55.

Conclusion

Docling and ColPali answer the same question in opposite ways: Docling parses every page up front with six models, producing clean, searchable markdown at the cost of slow, GPU-heavy ingestion; ColPali skips parsing entirely, ingests 5.3x faster, and retrieves straight from the page image, at the cost of far more vectors per page and no text for the LLM. On our corpus, seeing beat parsing for retrieval, especially on charts, which no reranker can rescue once the parser has lost them. But the best answers came from combining the two: ColPali to find the pages, a light Docling pass to make them readable. Whichever you pick, read your chunks: every failure that mattered here was invisible in the aggregate metrics.