PDF RAG Pipeline
Grounded answers from PDFs, with citations that lead back to the document.
Overview
This multimodal retrieval-augmented generation pipeline connects PDF extraction to answer generation. Gemini handles extraction, while Qdrant and BM25 provide complementary retrieval methods. Fusion and reranking refine the retrieved context, and asynchronous workers generate answers with page and section citations.
Architecture
Scroll to explore the connections, or use Fit to see the full diagram.
Asynchronous PDF ingestion
- The client uploads a PDF to POST /upload. FastAPI creates a MongoDB file record, saves the PDF, and returns file_id after enqueueing the job.
- The Redis/Valkey queue carries the file ID and file path. Independent RQ workers pick up jobs and read the PDF from the uploads path.
- Workers convert pages with Poppler, extract content with Gemini, create chunks, and generate embeddings. MongoDB tracks progress and stores pages/chunks; Qdrant stores vectors and metadata.
Component responsibilities
FastAPI & job queue
POST /upload creates a MongoDB file record, writes the PDF to /mnt/uploads/, and enqueues the file ID and file path in Redis/Valkey. The client receives file_id after the job is queued.
RQ workers & document state
Worker processes read the saved PDF, rasterize its pages with Poppler, extract content with Gemini, and create chunks and embeddings. MongoDB stores progress, extracted pages, and chunks; Qdrant stores vectors and metadata.
Query & hybrid retrieval
POST /query combines Qdrant dense search with BM25 over MongoDB chunks, followed by Reciprocal Rank Fusion and cross-encoder reranking.
Context & answer generation
A token-budgeted context is assembled from the selected chunks. Gemini generates the answer, and FastAPI returns it with page, section, and chunk references.
Why I built this
I wanted to work through the full path from a complex PDF to an answer that could be checked against its source. The project let me study extraction, hybrid retrieval, reranking, and citation handling as connected parts of a RAG system.
Use case
Someone working through a long PDF can ask a question, retrieve relevant passages, and follow page or section citations to verify the answer in the original document.
What I learned
- Combining vector and keyword retrieval instead of relying on a single retrieval method.
- Using fusion and reranking to refine the context passed to generation.
- Separating asynchronous processing from API requests and preserving document references through the pipeline.
Key features
Multimodal extraction
Gemini extracts content from PDFs for use in the retrieval pipeline.
Hybrid retrieval
Qdrant and BM25 retrieval are combined with fusion and reranking to select relevant context.
Traceable answers
Generated answers include page and section citations, making the source material easier to check.

