focus on concept not only code
This tutorial walks you through the concepts behind building a Retrieval-Augmented Generation (RAG) system using completely free tools and models. Rather than just providing code, we'll explore why each component matters and how they work together.

Part 1: Understanding the Retrieval Pipeline

Before we write any code, let's understand what we're building and why each piece exists.

---

1.1 What is RAG and Why Does It Matter?

The Problem with Traditional LLMs

Imagine asking a general AI assistant:

"What were the key findings from our Q3 marketing report?"

A standard Large Language Model (LLM) would typically:

• ❌ Hallucinate a convincing but incorrect answer.
• ❌ Refuse because it doesn't have access to your private documents.
• ❌ Provide a generic response that isn't based on your actual data.

The RAG Solution

Retrieval-Augmented Generation (RAG) solves this problem by allowing the LLM to search your documents before generating an answer.

Think of it as giving an expert access to a reference library immediately before they respond.

Traditional LLM:
"I know some things, but not everything."

RAG System:
"I'll search your documents first, then answer based on the information I find."

---

1.2 The Two Critical Phases of RAG

Every RAG system works in two separate phases.

Think of it like preparing a library and then helping someone find the right books.

Phase 1: Indexing (Preparation)

This step is performed once before users start asking questions.

• Import all documents.
• Split documents into smaller chunks.
• Convert each chunk into vector embeddings.
• Store everything in a vector database for fast retrieval.

Phase 2: Querying (Real-Time)

This process happens every time a user asks a question.

• Convert the user's question into an embedding.
• Search the vector database for similar document chunks.
• Retrieve the most relevant chunks.
• Pass those chunks to the LLM as context.
• Generate a grounded answer.

---

1.3 Why Free Models Work Perfectly for RAG

Many people assume RAG requires expensive AI models.

In reality, free open-source models are more than capable for most RAG systems.

Task	Why Free Models Work	Recommended Free Option
Embedding	Semantic understanding is more important than model size.	BAAI/bge-small-en-v1.5 (Runs locally)
Retrieval	Vector similarity search is computationally inexpensive.	DuckDB + Vector Extension
Generation	The LLM only needs to interpret retrieved context, not memorize everything.	Groq Llama 3.3 70B, Gemini Flash, DeepSeek R1

---

1.4 The Embedding Concept: Turning Text into Numbers

Embeddings are one of the most important concepts in Retrieval-Augmented Generation.

Instead of storing plain text, we convert each piece of text into a mathematical vector.

Text:
"The cat sat on the mat"

        ↓

Vector:
[0.23, -0.45, 0.67, 0.12, ...]
(384 floating-point numbers)

Why Convert Text into Numbers?

Computers are excellent at mathematics but cannot directly understand meaning.

Embedding models transform semantic meaning into numerical vectors.

This allows us to:

• Find semantically similar sentences.
• Compare meanings rather than exact words.
• Perform fast similarity searches.

Example

"The cat is sleeping."

"The kitten is taking a nap."

↓

Both generate vectors that are very close together.

"The stock market crashed."

↓

Produces a completely different vector.

Why Use BAAI/bge-small-en-v1.5?

This free embedding model:

• Creates 384-dimensional embeddings.
• Runs entirely on your local machine.
• Requires no API calls.
• Delivers excellent semantic search performance for most RAG applications.

---

1.5 Why Choose DuckDB as Your Vector Database?

Traditional databases are designed for exact text matching.

Vector databases are designed for semantic similarity search.

DuckDB combines both capabilities.

Traditional SQL

Search:
"marketing"

Returns only rows containing the exact word.

--------------------------------------------

Vector Search

Search:
"marketing strategy"

Can also retrieve:

• Advertising plan
• Sales campaign
• Customer acquisition roadmap

Even if those exact words aren't present.

DuckDB Retrieval Flow

Document Chunk A
        ↓
[0.23, -0.45, 0.67, ...]

Document Chunk B
        ↓
[0.89, 0.12, -0.33, ...]

Document Chunk C
        ↓
[0.45, -0.78, 0.22, ...]

-----------------------------

User Question
        ↓
[0.44, -0.76, 0.23, ...]

        ↓
Calculate vector similarity

        ↓
Find nearest vectors

        ↓
Return the most relevant document chunks

---

1.6 The Chunking Strategy: Size Matters

One of the most important design decisions in a RAG system is determining how documents should be split into chunks.

Why Split Documents?

• LLMs have limited context windows.
• Smaller chunks improve retrieval precision.
• Multiple chunks can capture different parts of the same document.

Chunk Size Comparison

Chunk Size	Advantages	Disadvantages
100–200 words	Highly focused and precise	Limited context
500–1000 words	Good balance between precision and context	May include some unrelated content
2000+ words	Rich context	Lower retrieval precision and slower searches

Recommended Strategy

For most RAG applications:

• Chunk size: 500 words
• Overlap: 100 words

Using overlap prevents important information from being lost at chunk boundaries.

---

1.7 The Retrieval Process: How Search Actually Works

Every user query follows the same retrieval pipeline.

User Question

"What are the quarterly targets?"

        ↓

Step 1
Convert the question into an embedding.

[0.34, -0.56, 0.78, ...]

        ↓

Step 2
Compare this vector with every document embedding.

        ↓

Step 3
Calculate cosine similarity.

Chunk A → 0.92
Chunk B → 0.87
Chunk C → 0.65

        ↓

Step 4
Select the top-ranked document chunks.

        ↓

Pass those chunks to the LLM as context.

        ↓

Generate the final answer.

Why Cosine Similarity?

Cosine similarity compares the angle between vectors rather than their raw distance.

Benefits:

• Works exceptionally well for high-dimensional embeddings.
• Measures semantic similarity rather than exact wording.
• Performs consistently across different document lengths.

As a result, the retrieval system can identify relevant content even when the user's question uses different wording than the original documents.

Part 2: Understanding the Generation Pipeline

Now that you've retrieved the most relevant document chunks, the next step is generating a high-quality answer. This phase transforms retrieved context into a grounded response using a Large Language Model (LLM).

---

2.1 The Prompt Engineering Concept

Retrieving relevant context is only half of the RAG pipeline. The way you present that context to the LLM has a significant impact on the quality and reliability of the generated answer.

❌ Poor Prompt

Here's some text:
[context]

Question:
[question]

Answer:

Problem:

• The model may ignore the provided context.
• It may rely on its own pre-trained knowledge.
• This increases the chance of hallucinations.

---

✅ Good Prompt

You are a helpful assistant.

ONLY use the provided context to answer the user's question.

If the answer is not present in the context, respond with:
"I don't have enough information in the provided documents."

Context:

[Chunk 1]

[Chunk 2]

[Chunk 3]

Question:
[question]

Answer based ONLY on the provided context:

Benefits

• Forces the model to use retrieved documents.
• Greatly reduces hallucinations.
• Produces grounded and trustworthy responses.

---

2.2 Why Multiple Free LLMs? Understanding Trade-offs

Different LLMs have different strengths. Choosing the right one depends on your application's requirements.

Groq Llama 3.3 70B — The Speed Demon

Strengths

• Extremely fast inference
• Excellent instruction following
• Large 70B parameter model

Weaknesses

• 30 requests/minute limit
• Slightly less creative than larger proprietary models

Best For

• Production applications
• Real-time chat
• Coding assistance
• High-speed responses

---

Google Gemini 1.5 Flash — The Volume King

Strengths

• 60 requests/minute
• Massive 1 million token context window
• Strong reasoning capabilities

Weaknesses

• Slower than Groq
• Subject to Google's safety restrictions

Best For

• High-volume applications
• Large document processing
• Long-context summarization

---

DeepSeek R1 — The Reasoning Expert

Strengths

• Excellent logical reasoning
• Chain-of-thought capabilities
• Open-source transparency

Weaknesses

• Accessed through OpenRouter
• Slightly slower response times

Best For

• Research
• Complex problem solving
• Analytical tasks

---

2.3 The Information Integration Problem

Retrieval usually returns multiple relevant document chunks.

Example:

Retrieved Chunks

1. Q1 Target: $1.2M
2. Q2 Target: $1.5M
3. Regional revenue breakdown
4. Historical sales performance
5. Product launch roadmap

Question:

"What are the annual targets?"

The Challenge

The answer isn't contained within a single chunk.

Instead, the model must:

• Read multiple chunks.
• Identify relevant information.
• Combine related facts.
• Produce one coherent response.

Recommended Solution

Provide retrieved chunks in descending order of relevance.

Most Relevant Chunk

↓

Supporting Information

↓

Additional Context

↓

Less Relevant Information

This ordering helps the LLM prioritize the most useful evidence first.

---

2.4 The Hallucination Prevention Mechanism

One of the biggest advantages of RAG is reducing hallucinations.

Without RAG

Question:
"What was Q3 revenue?"

Model:

"Q3 revenue was $2.1M."

⚠️ Completely fabricated.

---

With RAG

Question:
"What was Q3 revenue?"

Retrieved Document:

"Q3 revenue was $1.8M."

Model:

"According to the provided documents, Q3 revenue was $1.8M."

The Most Important Instruction

Always include a rule like this in your prompt:

If the answer is not present in the provided context,

respond with

"I don't have enough information in the provided documents."

This single instruction dramatically reduces hallucinations.

---

2.5 Why Automatic Model Selection Matters

Production systems should never rely on a single LLM provider.

A robust fallback strategy improves reliability.

Primary Model

Groq Llama 3.3

↓

Unavailable?

↓

Gemini Flash

↓

Unavailable?

↓

DeepSeek R1

↓

Unavailable?

↓

Return:

"Service temporarily unavailable."

Benefits

• Automatic failover
• Better uptime
• Rate-limit protection
• Transparent model switching

Users never need to know which model generated the response.

---

2.6 The Context Window Concept

Every LLM has a maximum amount of text it can process at once.

Model	Context Window
Groq Llama 3.3	128K Tokens
Gemini 1.5 Flash	1M Tokens
DeepSeek R1	128K Tokens

Typical RAG Prompt Size

5 Chunks

≈ 500 words each

↓

≈ 2,500 words

Question

≈ 50 words

Instructions

≈ 100 words

Total

≈ 2,650 words

This is far below the context limits of all three models.

Key Insight

A RAG system never sends the entire knowledge base to the model.

Instead, it sends only the most relevant snippets.

That's why even free-tier models perform exceptionally well.

---

2.7 The Cost-Performance Sweet Spot

RAG significantly reduces AI costs.

Without RAG

Need expensive models

↓

Model must memorize everything

↓

Higher hallucination risk

---

With RAG

Retrieve relevant documents

↓

Feed only useful context

↓

Use smaller free models

↓

Generate grounded answers

Cost Comparison

Enterprise RAG

≈ $0.01–$0.10 per query

----------------------------

Free RAG

≈ $0.00 per query

Using:

• Groq Free Tier
• Gemini Free Tier
• Local Embeddings

---

2.8 Real-World RAG Patterns

Pattern 1 — Question Refinement

User:

"Tell me about the project."

↓

System Reformulates

"What is the project's purpose and key features?"

↓

Improved retrieval quality.

---

Pattern 2 — Multi-Hop Reasoning

Question

"What caused the drop in sales?"

↓

Retrieve

• Sales reports
• Competitor analysis
• Market trends

↓

LLM synthesizes

"Sales decreased by 15% because competitors lowered prices..."

---

Pattern 3 — Filtering and Ranking

Retrieve

10 Chunks

↓

Rank by similarity

↓

Keep Top 5

↓

Generate final answer

This approach improves both precision and response quality.

---

2.9 Practical Design Decisions

Storage Strategy

Option	Advantages
Local Storage	Private, secure, no API costs
Cloud Storage	Scalable and accessible

---

Document Updates

Document Type	Strategy
Static	Index once
Frequently Updated	Periodic re-indexing
Live Data	Real-time indexing

---

Retrieval Strategy

Strategy	Description
Fixed	Always retrieve Top 5 chunks
Adaptive	Adjust chunk count based on question type
Dynamic	Retrieve additional context for complex queries

---

2.10 Understanding the Limitations

Every RAG system has limitations.

What RAG Can Do

• Answer questions about your documents.
• Generate grounded responses.
• Work with private or proprietary information.
• Reduce hallucinations.

What RAG Cannot Do

• Answer questions beyond the document collection.
• Invent missing knowledge.
• Reliably solve highly complex multi-step reasoning tasks without sufficient context.

Reality Check

RAG

≈ Search Engine

+

Summarizer

----------------------------

RAG

≠

Omniscient Artificial Intelligence

---

Summary: The Complete RAG Pipeline

User Question

        ↓

Embed Query

(Convert question into vectors)

        ↓

Vector Search

(Find similar document embeddings)

        ↓

Retrieve Context

(Top 5 most relevant chunks)

        ↓

Build Prompt

(Instructions + Context + Question)

        ↓

Generate Answer

(Groq / Gemini / DeepSeek)

        ↓

Grounded Response

(Based entirely on retrieved documents)

---

Key Takeaways

• ✅ Free LLMs are powerful enough for production-quality RAG systems.
• ✅ High-quality retrieval is more important than using the largest LLM.
• ✅ Prompt engineering directly affects answer quality.
• ✅ Supporting multiple LLM providers improves reliability and availability.
• ✅ RAG dramatically reduces hallucinations by grounding responses in retrieved documents.
• ✅ With local embeddings and free API tiers, the operational cost can be effectively zero.
• ✅ Local embedding models help preserve privacy because your documents remain on your own system.

The strength of RAG lies in its simplicity and transparency. Rather than expecting an LLM to memorize everything, a RAG system retrieves relevant information from your knowledge base and then asks the model to summarize or explain it. This retrieval-first architecture has become the standard approach for enterprise AI applications because it is accurate, scalable, cost-effective, and significantly more trustworthy than relying on an LLM alone.