Building RAG Systems at Scale: Lessons from Production
Why RAG Over Fine-Tuning
When a client asked me to build an enterprise knowledge base, the first instinct was to fine-tune a model on their internal documents. But fine-tuning has serious drawbacks for enterprise use: it bakes knowledge into model weights (making updates expensive), it hallucinates when asked about information outside the training set, and it cannot cite sources. RAG solves all three problems by keeping the LLM as a reasoning engine and fetching relevant context at query time from a vector database.
The Chunking Problem
The single biggest factor in RAG accuracy is how you chunk your documents. I started with naive 500-token chunks and got mediocre results. The breakthrough came from implementing semantic chunking — splitting documents at natural boundaries (headings, paragraph breaks, topic shifts) rather than at arbitrary token counts. Combined with overlapping windows of 50 tokens, retrieval accuracy jumped from 72% to 94%. For tables and structured data, I built a separate extraction pipeline that preserves relationships before embedding.
Embedding Strategy
We used OpenAI's text-embedding-3-large for production embeddings stored in Pinecone. But the real performance gain came from hybrid search — combining dense vector similarity with sparse BM25 keyword matching. Some queries are better served by exact keyword matches (product codes, policy numbers), while others need semantic understanding. A weighted combination of both consistently outperformed either approach alone.
Latency Optimization
Our target was sub-500ms end-to-end response time. The naive approach (embed query → search Pinecone → pass results to GPT-4) took 3-4 seconds. We got it under 500ms with three changes: pre-computing and caching frequent queries with Redis, streaming the LLM response so the user sees tokens immediately, and using GPT-3.5-turbo for simple factual lookups while routing complex reasoning queries to GPT-4.
What I Would Do Differently
If I were starting this project today, I would invest more time upfront in evaluation infrastructure. We built our test suite after deployment and discovered accuracy issues that could have been caught earlier. I would also explore open-source embedding models — the quality gap with OpenAI has narrowed significantly, and self-hosting eliminates a dependency and reduces cost at scale.