• PII Data RAG Pipeline
PII Data RAG Pipeline
A document question-answering project built with LangChain, Hugging Face embeddings, Pinecone, and Groq. It retrieves relevant PDF text and uses it as context for answers about invoices, purchase orders, shipping records, and inventory reports.
Project Type
Development, Data Engineering, Machine LearningTools Used
Python, LangChain, Hugging Face, Pinecone, Groq, Jupyter NotebookProject Overview
This notebook explores retrieval-augmented generation (RAG) using a sample collection of company PDFs. It creates searchable embeddings from document text, retrieves the closest matches to a question, and passes that context to a language model. The focus is document retrieval and answer generation; PII detection and redaction are outside the current implementation.
Key Features
- •Natural-language questions about company records
- •Semantic search across indexed PDF text
- •Source filenames and folders retained as metadata
- •Answers generated using retrieved document context
Technical Highlights
- •PyPDFLoader reads the PDFs; the notebook indexes the first page of each file
- •all-MiniLM-L6-v2 creates 384-dimensional document and query embeddings
- •Pinecone stores the vectors and returns the ten closest matches per query
- •Groq runs Llama 3.1 with the retrieved text included in the prompt
Vector Embedding Space
A conceptual illustration of how document text can be represented as vectors for similarity search.
Query Process Flow
The question is embedded with the same model. Pinecone returns the ten closest matches, and their text is combined with the question for a response from Groq-hosted Llama 3.1.
Pipeline Architecture
A high-level view of document processing, embedding generation, vector storage, and answer generation.
Next Project
