Python RAG Chatbot

Overview

Python RAG Chatbot is a document Q&A system built on Retrieval-Augmented Generation (RAG) — it answers questions about a set of documents by retrieving the semantically relevant chunks and passing them to an LLM as context, rather than letting the model answer from memory. What started as a fully local tool has evolved into a live deployed microservice: a FastAPI backend on Railway, wired directly into my portfolio as a floating AI assistant.

Problem

A portfolio site is read-only by default — a recruiter has to manually dig through project pages to find the answer to a specific question ("did this use SQL or NoSQL?", "what was the biggest technical challenge?"). Generic chatbots don't solve this either, since a plain LLM will confidently hallucinate details it was never given. This project solves both: it lets a visitor ask a direct question and get an answer grounded specifically in my actual project documents, with the retrieval step making sure the model can't just make things up.

Tech Stack

• API framework: FastAPI + Uvicorn
• RAG pipeline: LangChain
• Vector store: ChromaDB
• Embeddings: HuggingFace all-MiniLM-L6-v2
• LLM: Llama 3.2 via Groq API
• Deployment: Railway (Python service, auto-deploy from GitHub)
• Integration: Next.js API route proxy → portfolio chat widget

Technically Interesting

The RAG pipeline itself:

1. Ingestion — documents are split into overlapping chunks using LangChain's text splitter (chunk size 500, overlap 50), so context isn't lost at chunk boundaries.
2. Embedding — each chunk is embedded with HuggingFace's all-MiniLM-L6-v2 into a 384-dimensional vector.
3. Storage — vectors are persisted on-disk in ChromaDB.
4. Retrieval — the user's query is embedded and cosine-similarity matched against stored vectors to pull the top-k relevant chunks.
5. Generation — those chunks are injected into the Llama 3.2 prompt as context via Groq, so the model answers from the retrieved text, not its training data.

The architecture: the visitor's browser never talks to Railway directly. A Next.js /api/chat proxy route sits in between, so the Railway URL and all external API calls stay server-side:

Visitor → Next.js /api/chat proxy → FastAPI (Railway) → LangChain RAG pipeline
                                                          ↙              ↘
                                                     ChromaDB          Groq API
                                                   (vector store)   (Llama 3.2)
                                                          ↘              ↙
                                                        streamed answer back

Local-to-production tradeoff: the original version ran fully offline via Ollama, by design — documents never left the machine. Moving to production meant swapping Ollama for Groq's cloud inference, trading that local-only privacy guarantee for meaningfully faster response times. For a portfolio use case where the documents are just my own project descriptions, that tradeoff was the right call.

Why FastAPI: Python-native, async-friendly, ships automatic Swagger docs at /docs, and deploys to Railway with a single Procfile — the /chat endpoint takes a POST with a question field and returns the LLM's answer as JSON.

Why Groq over OpenAI: Groq serves the same open-source Llama 3.2 model at meaningfully faster inference speeds, with a free tier generous enough for portfolio traffic — and it demonstrates that a working AI feature doesn't require an expensive proprietary API.

Result

A live, deployed RAG microservice, not just a local script:

• Deployed at: pythonrag-production.up.railway.app, integrated into joshuario.com as a floating chat widget
• Response time: typically under 2 seconds per query via Groq
• Format support: PDF, DOCX, TXT, and XLSX ingestion
• A visitor can open the chat widget on the live portfolio and ask direct questions about any project, getting answers grounded in the actual project write-ups instead of generic LLM guesses

What I'd Improve

• Add streaming responses so the answer types out character by character instead of arriving all at once
• Add conversation memory so follow-up questions have context from previous turns
• Add document upload via the API so users can query their own files, not just my pre-loaded project descriptions
• Move ChromaDB to a persistent cloud store (Pinecone or Qdrant) so the vector index survives Railway restarts