Multimodal RAG: Retrieval Over PDFs and Tables

Short answer: most enterprise documents carry their meaning in tables, charts and layout, and a text-only pipeline throws that away at ingest. Multimodal RAG indexes the visual content too — either by parsing structure properly or by embedding page images directly. The parsing route is cheaper and better when it works; the image route is the fallback when structure resists extraction.

The failure this fixes is quiet. Retrieval returns something plausible, the answer is wrong, and nobody connects it to a table that was flattened into gibberish six weeks earlier.

What text-only extraction destroys

Run a financial report or a technical manual through naive text extraction and watch what happens.

Tables collapse. A table is a two-dimensional relationship between headers and cells. Flattened to a line of text it becomes a sequence of numbers with no indication of which column they belonged to. The model then confidently attributes Q3 revenue to Q2.

Charts vanish. A revenue trend expressed as a line chart extracts as an axis label and a caption, if anything. The information was in the shape.

Layout meaning is lost. Multi-column pages interleave. Sidebars merge into body text. Footnotes attach to the wrong sentence. A form’s labels detach from its fields.

Figures with embedded text disappear. Architecture diagrams, flowcharts, annotated screenshots — often the densest information on the page.

If your corpus is prose, none of this matters and text-only is correct. If it is reports, manuals, financial filings or scientific papers, you are discarding the answer at ingest.

Recovering what flat text extraction throws away is a large share of the ingestion work in document-heavy RAG builds.

Two architectures

Parse then index. Use a document AI pipeline to recover structure — detect tables and reconstruct them as rows and columns, extract figure captions, preserve reading order, keep headings as hierarchy. Then index that structured representation as text, with tables serialised in a form that survives chunking, such as markdown with headers repeated per chunk.

This is our default. It is cheaper at query time, the retrieved context is readable and citable, and the structure is available downstream for anything else you want to do with it. It depends entirely on extraction quality, which is the document AI problem rather than the retrieval problem.

Embed page images. Use a vision-language model to embed rendered page images directly, retrieve pages by visual similarity, and pass the images to a multimodal model at generation time. Nothing is lost because nothing is converted.

It costs more per query, retrieval is coarser — you get a page rather than a passage — and citations point at pages rather than sentences. But it works on documents that resist parsing entirely: scanned forms, dense layouts, heavy diagrams.

In practice, use both. Parse what parses cleanly, fall back to page images for the document types where extraction quality is poor. Classify at ingest and route accordingly — the same per-type discipline that governs chunking.

Tables deserve their own treatment

Tables are the highest-value and most commonly broken element, and they need specific handling rather than a general strategy.

Never split a row across chunks. Half a row is worse than no row, because it retrieves and looks answerable.

Repeat the header in every chunk. A chunk of rows without column names is unusable. This costs tokens and is not optional.

Keep a text summary alongside the structure. Embedding a serialised table often retrieves poorly, because the vector is dominated by numbers. Generating a one-line description of what the table contains at ingest — and embedding that for retrieval, while returning the actual table to the model — dramatically improves matching. This is the single highest-leverage trick in multimodal RAG.

Preserve the caption and surrounding sentence. They usually say what the table means, which the table itself does not.

Chunking changes shape

The clean split for text — recursive or semantic — does not apply to mixed content. What works is chunking on document structure: a section, a table, a figure with its caption. Each becomes a unit with its type recorded as metadata.

That metadata pays off at retrieval. A question about a number can bias toward table chunks; a question about a process can bias toward figures. Filtering by chunk type before ranking is cheap and noticeably improves precision.

Evaluate it separately

Aggregate metrics hide this class of failure completely. A pipeline can score well overall while getting every table question wrong, because tables are a minority of chunks.

Segment the golden set by content type — questions answered by prose, by tables, by figures — and track recall for each. The table number is usually the worst and the most fixable, and you will not see it in an average.

When not to bother

Multimodal adds real cost — extraction infrastructure, larger indexes, sometimes a vision model at query time. Skip it when your corpus is genuinely prose, when the tables are decorative rather than load-bearing, or when the underlying data behind those tables is available in a database you could query directly.

That last one is worth pausing on. If a table in a PDF is a rendering of data you already hold, retrieval over the PDF is the wrong architecture. Query the source and let the model use structured results. Teams build elaborate table extraction to recover data they already own more than you would expect.

The takeaway

Check what your ingest pipeline does to a representative table before assuming text-only is fine — that one test tells you whether you have this problem. Parse where parsing works, fall back to page images where it does not, treat tables as first-class with repeated headers and summary embeddings, and measure retrieval by content type rather than in aggregate.

EpochC provides custom RAG development services and OCR and document AI, including table structure recognition and layout-aware extraction. See the KYC OCR automation case study, or start a project.

Multimodal RAG: retrieval over PDFs, tables and diagrams