Multimodal RAG Assistant over an Internal Knowledge Base

0) Problem Restatement

Apple asked: design a Retrieval-Augmented Generation (RAG) assistant that answers employee questions using an internal knowledge base containing text, images, PDFs and tables. Answers must be grounded in the documents (with citations), respect access permissions, and stay up to date as documents change.

1) Architecture

flowchart LR
    SRC["Docs, wikis, PDFs, images"] --> ING["Ingestion - parse, OCR, caption, chunk"]
    ING --> EMB["Embeddings - text + image"]
    EMB --> VDB[("Vector index + keyword index + ACLs")]
    U["User question"] --> QS["Query service"]
    QS --> VDB
    QS --> RR["Re-ranker"]
    RR --> LLM["LLM - answer with citations"]
    LLM --> U
    QS --> LOG[("Logs + feedback")]

2) Ingestion (the most important part)

• Parse each format:
• Text and HTML → clean text with headings.
• PDFs → text extraction, plus OCR for scanned pages, plus layout-aware parsing to keep tables and sections.
• Tables → keep them as structured rows (or markdown), and create a short description of what the table contains.
• Images (diagrams, screenshots) → generate a caption with a vision model, extract text in the image (OCR), and create an image embedding (e.g., CLIP-like) so image-to-text search works.
• Chunking: split into meaningful pieces (~300–800 tokens), keeping headings and page numbers, with a small overlap. Each chunk keeps metadata: doc ID, section, page, modality, ACL (who can see it), and updated time.
• Freshness: a change feed from sources re-indexes changed documents and deletes removed ones.

3) Retrieval

1. Understand the query: rewrite it (expand acronyms, add conversation context).
2. Hybrid search: vector similarity (meaning) + keyword/BM25 (exact terms like product codes), with results merged.
3. Permission filter: only chunks whose ACL includes the user (their groups), applied inside the search, not after, to avoid leaks and empty results.
4. Re-rank the top ~50 with a cross-encoder model and keep the best ~5–10. Include images or tables when they're relevant (e.g., a diagram caption matched).

4) Generation

• The prompt contains the question plus the retrieved chunks (with IDs), and the instructions: answer only from the context, cite sources, say "I don't know" if the context lacks the answer.
• For images, pass the image itself to a multimodal model when needed (e.g., "what does the architecture diagram show?"), or its caption.
• Return the answer with citations that link to the document and page.

5) Quality, Safety and Scale

• Evaluation: a test set of real questions with expected sources. Measure retrieval recall@k (did we find the right chunk?), groundedness (is every claim supported?), and answer correctness. Track thumbs up/down in production.
• Hallucination control: citations required, low-confidence answers flagged, and grounding checks before display.
• Latency: embedding the query (~20 ms) + search (~50 ms) + re-rank (~100 ms) + LLM (~1–3 s, streamed). Cache frequent questions.
• Scale: millions of chunks → an ANN vector index (HNSW) sharded by tenant or department.

6) Wrap-Up

Ingest every format properly: text, PDFs (with OCR and layout), tables as structured text with summaries, and images via captions, OCR and image embeddings. Chunk with metadata and ACLs, and keep the index fresh from change feeds. Answer with hybrid search filtered by the user's permissions, cross-encoder re-ranking, and an LLM prompted to answer only from the cited context (passing images to a multimodal model when needed), and measure retrieval recall and groundedness continuously.