PROJECT / 02Jul 2026 — Aug 2026

PDF RAG Pipeline

Grounded answers from PDFs, with citations that lead back to the document.

PDF extraction, retrieval, and answer pipeline
PDF extraction, retrieval, and answer pipeline

Overview

This multimodal retrieval-augmented generation pipeline connects PDF extraction to answer generation. Gemini handles extraction, while Qdrant and BM25 provide complementary retrieval methods. Fusion and reranking refine the retrieved context, and asynchronous workers generate answers with page and section citations.

Architecture

System architecture
100%

Scroll to explore the connections, or use Fit to see the full diagram.

Asynchronous PDF ingestionFastAPI accepts a PDF, returns a file ID, and queues an ID plus file path. RQ workers process the document, update MongoDB, and index vectors in Qdrant.ClientPDF upload / file_idFastAPI serverPOST /uploadMongoDBfiles: status + pageschunks: text + sourcesRedis / ValkeyRQ queue · ID + pathRQ worker poolworker 1worker 2worker nPDF storage/mnt/uploads/PDF files + page imagesInside each workerPoppler → GeminiChunks → embeddingsQdrantVectors + metadata1 · Upload PDF2 · Return file_idSave file statusSave PDFEnqueue jobDequeueRead PDFRun pipelineProgress + pages + chunksIndex embeddings

Asynchronous PDF ingestion

  1. The client uploads a PDF to POST /upload. FastAPI creates a MongoDB file record, saves the PDF, and returns file_id after enqueueing the job.
  2. The Redis/Valkey queue carries the file ID and file path. Independent RQ workers pick up jobs and read the PDF from the uploads path.
  3. Workers convert pages with Poppler, extract content with Gemini, create chunks, and generate embeddings. MongoDB tracks progress and stores pages/chunks; Qdrant stores vectors and metadata.

PDF RAG · System architecture

Component responsibilities
  1. FastAPI & job queue

    POST /upload creates a MongoDB file record, writes the PDF to /mnt/uploads/, and enqueues the file ID and file path in Redis/Valkey. The client receives file_id after the job is queued.

  2. RQ workers & document state

    Worker processes read the saved PDF, rasterize its pages with Poppler, extract content with Gemini, and create chunks and embeddings. MongoDB stores progress, extracted pages, and chunks; Qdrant stores vectors and metadata.

  3. Query & hybrid retrieval

    POST /query combines Qdrant dense search with BM25 over MongoDB chunks, followed by Reciprocal Rank Fusion and cross-encoder reranking.

  4. Context & answer generation

    A token-budgeted context is assembled from the selected chunks. Gemini generates the answer, and FastAPI returns it with page, section, and chunk references.

Why I built this

I wanted to work through the full path from a complex PDF to an answer that could be checked against its source. The project let me study extraction, hybrid retrieval, reranking, and citation handling as connected parts of a RAG system.

Use case

Someone working through a long PDF can ask a question, retrieve relevant passages, and follow page or section citations to verify the answer in the original document.

What I learned

  • Combining vector and keyword retrieval instead of relying on a single retrieval method.
  • Using fusion and reranking to refine the context passed to generation.
  • Separating asynchronous processing from API requests and preserving document references through the pipeline.

Key features

  1. Multimodal extraction

    Gemini extracts content from PDFs for use in the retrieval pipeline.

  2. Hybrid retrieval

    Qdrant and BM25 retrieval are combined with fusion and reranking to select relevant context.

  3. Traceable answers

    Generated answers include page and section citations, making the source material easier to check.

More projects

View all
GitHub
LinkedIn
X