RAG Super Tutorial: From Principles to Practice – Building Accurate and Cost-Effective AI Question Answering Systems Using Vector and Reordering Models

Have you ever had the urge to create an “in-site AI assistant” for a website with hundreds of articles? You’d simply copy the entire text of all the articles into the system prompt and then ask your question, right?

That’s exactly what I did at first. It wasn’t until the bill came in, the answers started making no sense, and the long articles exceeded the context window that I realized this was a dead end.

This tutorial explains RAG (Retrieval-Augmented Generation) in detail: **why you shouldn’t just “stuff everything into the prompt,” what the correct approach is, and how to use two free models from SiliconFlow (BAAI/bge-m3 and BAAI/bge-reranker-v2-m3) to build an accurate and cost-effective question-answering system.

If you want to use these two free models for free, you can use this registration link: https://cloud.siliconflow.cn/i/MBwLMSrY

---

I. What is RAG (Why)

RAG = Retrieval-Augmented Generation: Before asking a large model to generate an answer, it first retrieves the most relevant information from an external knowledge base and provides that information along with the question to the model, allowing it to answer with that context in mind.

The core idea is simple: let the model look up the information before answering, rather than expecting it to have all the knowledge in its parameters.

Why did RAG emerge?

Large models have several inherent limitations:

• Knowledge has an expiration date: Models only “know” what was available at the time of training; anything that happened after that is unknown to them.
• They can’t access your private data: Company documents, your blog posts, customer chat records—none of this is included in their training data.
• The context window is limited: Even with a large context window of 200,000 tokens, it’s impossible to fit an entire knowledge base.
• They might generate incorrect answers (hallucinations): The model might fabricate details even if the relevant information isn’t available.

The paper “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” published by Meta in 2020, proposed using RAG to help models look up information before answering, thereby addressing these issues. As vector databases and embedding models have matured, RAG has become the most popular approach for implementing large language model (LLM) applications.

What problems does it solve / How does it differ from traditional large model answers?

	Traditional Large Model Answers	RAG-Enhanced Answers
Source of Knowledge	Parameters learned during training	Real-time retrieved information from external sources
Timeliness	Stays at the time of training	Updates as new information becomes available
Access to Private Data	Unable to access	Can retrieve private company data
Traceability	Can’t provide sources	Can cite the original information
Accuracy	Prone to hallucinations	Limited by the available information

In short: Traditional large models are like taking an exam from memory, while RAG is like having open-book access to information.

---

II. Why Do We Need RAG?

Let’s focus on the main challenges, which are also the reasons why RAG exists:

1. Large model knowledge becomes outdated: Models from 2024 might not be able to answer questions about new policies or APIs from 2026.
2. Hallucinations: Models might confidently fabricate facts, links, or numbers when there’s no relevant information.
3. Lack of access to private data: Models have no access to internal wikis, contracts, or work orders.
4. Long prompts and high token costs: Copying all website articles into the prompt results in millions of tokens being used for each question.
5. Lack of citation: Users can’t verify the source of information, leading to distrust.

RAG separates the process of “memorizing” from “looking up information”: The model is responsible for reasoning and expression, while the knowledge is stored externally and can be updated at any time.

III. The RAG Workflow

A flowchart can explain this process clearly:

flowchart LR
  Q[User Question] --> E[Embedding Vectorization]
  E --> V[Vector Database]
  V --> R[Retriever (Retrieve Top-K Results)]
  R --> RR[Reranking (Optional)]
  RR --> L[LLM (Generate Answer)]

In text, this looks like:

User Question
   ↓
Embedding (Vectorization: Converts the question into a vector)
   ↓
Vector Database (Stores vectors of all content)
   ↓
Retriever (Retrieves the top-K segments most similar to the question vector)
   ↓
Reranker (Optional: Re-ranks the retrieved results based on relevance)
   ↓
LLM (Generates an answer using the most relevant segments)

Function of each step:

• Embedding: Converts natural language into a sequence of numbers (a vector), allowing for the measurement of semantic similarity through mathematical distances.
• Vector Database: Stores vectors of all content, enabling efficient searches for the closest neighbors based on these vectors.
• Retriever: Uses the question vector to perform a nearest neighbor search in the database and identifies a preliminary list of candidates (e.g., 15 segments).
• Reranker: Re-ranks the candidates by combining the question with each segment to determine the most relevant ones (e.g., 5 segments). This is a critical step for quality.
• LLM: Processes only the top-N segments to generate an answer with citations.

---

## IV. Core Components of RAG

### 1. Embedding Model

This model maps text into dense vectors of a fixed dimension. Common dimensions include 768, 1024, or 1536. For Chinese contexts, **BAAI/bge-m3** (1024 dimensions) is preferred as it performs well for both Chinese and English, and it is **free** on silicon-based platforms.

**What exactly is a “vector”?**

In simple terms, a vector is a sequence of numbers that represents the position of a piece of text in a “semantic space.” Imagine a map where each article or sentence is a point; semantically similar sentences are located near each other, while unrelated ones are far apart. Since it’s not possible to directly calculate the “distance” between text, embedding models convert the text into a fixed-length sequence of numbers, such as `[0.12, -0.03, 0.87, …]`. This sequence of numbers represents the “coordinates” of that sentence.

- Higher dimensions can capture more subtle nuances in semantics (e.g., 1024 dimensions can represent more details than 384 dimensions).
- “Semantic similarity” is roughly equivalent to “close vector distances”: It is measured using **cosine similarity** (the cosine of the angle between two vectors); a value closer to 1 indicates greater relevance, and closer to 0 indicates less relevance.

**A common misconception: Do large models process vectors directly?**

No. It’s important to distinguish between the two processes:
1. **Vectors are only used for retrieval**: During the retrieval phase, the system converts the question into a vector and searches the vector database for the closest text segments. This entire process is purely mathematical (calculating distances and sorting), and large models are not involved.
2. **The actual input to the large model is the original text**: After retrieval, the system combines the original text segments with the vector to create a prompt for the large model. The large model then processes this text to generate the answer.

Therefore, a more accurate description is: **Vectors help in finding the right segments, while the large model understands the meaning of the text.** Vectors serve as “addresses” for the retriever, not the content for the large model to process. This division of labor allows RAG to use cost-effective models to provide accurate answers, as the large model only sees the carefully selected segments.

### 2. Chunking

Long documents are divided into smaller segments. The retrieval granularity in RAG is based on these chunks, not the entire document. If the chunks are too large, the retrieval may not be precise; if they are too small, the semantics may be lost. A common size range is **300–500 characters**.

### 3. Vector Database

A database dedicated to storing vectors that supports approximate nearest neighbor (ANN) searches. Options include Qdrant, Milvus, Chroma, or **pgvector** (an extension for Postgres), which is used in this blog.

### 4. Retriever

This component is responsible for the initial screening of “question vector → candidate chunks.” The main algorithms used for this step are cosine similarity, inner product, or Euclidean distance.

### 5. Reranker

The Reranker uses a Cross-Encoder architecture, which is much more accurate than simple cosine similarity methods. It scores the combination of the query and each candidate segment. **BAAI/bge-reranker-v2-m3** is also **free** on silicon-based platforms.

### 6. LLMs (Language Generation Models)

The top-N results generated are used to form the final answer. Since only a few segments are displayed, it is possible to use a cheaper model, which significantly reduces costs.

How they work together: The **embedding model** is responsible for vectorization during data storage and retrieval; the **vector library + retriever** handles the preliminary filtering; the **reordering model** performs the fine-tuning; and the **LLM** is responsible for generating the actual text. The first three components ensure the accuracy of the findings, while the last one determines the quality of the answer.

---

## The Advantages of RAG (Retrieval-Augmented Generation)

- **More accurate answers**: Answers are based on real data, not fabricated from scratch.
- **Reduced likelihood of misinformation**: The scope for fabrication is limited due to the constraints imposed by the available data.
- **Credible sources**: Each answer includes a reference to the source material, allowing users to verify the information.
- **Real-time knowledge updates**: New documents can be added to the database without the need to retrain the models.
- **Lower token consumption**: Only the top-N segments are returned (which require a few thousand tokens), not the entire database.
- **The use of cheaper generation models is possible**: Even smaller models perform well since the context is brief and concise.
- **No need to retrain corporate-specific models**: The knowledge is external to the models, so their parameters remain unchanged.

## VI. The Difference Between RAG and Fine-tuning

This is a frequently searched topic, and the following table clarifies the differences:

| Dimension | RAG | Fine-tuning |
|---|---|---|
| Knowledge Update | Fast (effective immediately with additional documentation) | Slow (requires retraining) |
| Whether Model Parameters Are Changed | No change to parameters | Modification of model weights |
| Real-time Knowledge Update | Supported | Not supported; requires retraining |
| Cost | Low (mainly retrieval + minimal generation) | High (training power + data annotation) |
| Suitable Scenarios | Knowledge bases / retrieval-based question answering | Teaching the model specific “abilities” or “styles” |

> **Key Point**: Many companies actually use a combination of **RAG and fine-tuning** – RAG provides the latest, traceable knowledge, while fine-tuning helps the model adopt a particular expression style or task format. It’s not an either/or choice.

---

## VII. How to Set Up a RAG System

The complete process is as follows:

1. **Prepare Documents**: Gather original materials such as blog posts, PDFs, wikis, and tickets.
2. **Text Chunking**: Split the text into segments of 300–500 words each based on semantics.
3. **Embedding**: Convert each segment into a vector using bge-m3.
4. **Store in a Vector Database**: Store the segments, vectors, and metadata together in the database.
5. **User Query**: A question is received.
6. **Retrieval**: The question is vectorized, and the top 15 results are retrieved from the database.
7. **Reordering (Optional but Recommended): Use bge-reranker to reorder the results and select the top 5.
8. **Pass to the LLM for Answer Generation**: The top 5 results are combined with a prompt to generate an answer, along with the source information.

---

## VIII. Common Open-source Solutions (Ecosystem Overview)

**Frameworks**:
- LangChain (most popular, with many components)
- LlamaIndex (strengths in data integration and indexing)
- Haystack (enterprise-level pipeline)

**Embedding Models**:
- BGE (e.g., BAAI/bge-m3, strong for both Chinese and English, free for silicon-based systems)
- Jina Embeddings
- Nomic Embed

**Vector Databases**:
- Qdrant (written in Go, high performance, easy to deploy)
- Milvus (large-scale distributed)
- Chroma (lightweight, user-friendly for development)
- FAISS (Meta’s vector retrieval library, often used as the underlying engine)

**ReRankers**:
- BGE Reranker (BAAI/bge-reranker-v2-m3, free)
- Jina Reranker

---

## IX. How to Optimize RAG (Easiest to Overlook, but Most Valuable)

1. **Choosing the Right Chunk Size**: A segment size of 300–500 words is generally optimal; for longer documents, you can first split them by headings or paragraphs.
2. **Whether to Use Overlap**: It’s recommended to have 10%–20% overlap between segments to prevent semantic loss due to segmentation.
3. **Hybrid Search**: Combining vector retrieval (semantic) with keyword retrieval (BM25/full-text indexing) (RRF) significantly improves recall. This blog uses a dual RRF approach that combines vector and ilike lexical information.
4. **Metadata Filtering**: Filter results by time, category, or author before retrieval to reduce noise (e.g., searching for “tutorials from 2026”).
5. **Setting the Top-K**: Setting the recall to 15–20 and then selecting the top 5 after reordering is a safe starting point; too many results can introduce noise.
6. **The Need for a Reranker**: A reranker is almost always necessary as it can correct incorrect rankings and significantly improve the quality of the answers.
7. **Reducing Token Costs**: Only send the top 5 results after reordering; use inexpensive generation models; for embedding and reordering, use free models.
8. **Improving Recall**: Combine hybrid retrieval, metadata filtering, proper chunking, and reordering for optimal results.

---

## X. Use Cases for RAG

- **Enterprise Knowledge Bases**: Employees can ask about policies, procedures, and product documentation.
- **AI Customer Service**: Retrieve historical tickets and FAQs to provide accurate responses.
- **PDF-based Q&A**: Users can upload contracts, papers, or manuals and ask questions directly about the content.
- **API Documentation Assistance**: Developers can ask how to use an API and get the relevant sections.
- **Legal Consultation**: Retrieve legal statutes and case studies (which must be manually reviewed and not relied on solely).
- **Medical Knowledge Assistance**: Search for guidelines and literature (to assist, not replace professional judgment).
- **Educational Platforms**: Provide answers to questions based on textbook chapters and generate practice questions.
- **Blog Q&A**: Similar to this blog, where users can ask questions about specific articles, and the model answers based on those articles.

---

## XI. Common Misconceptions

- **RAG ≠ Fine-tuning**: RAG provides knowledge, while fine-tuning teaches the model new abilities; do not confuse the two.
- **RAG Does Not Make the Model Smarter**: RAG merely provides the data; the model’s capabilities remain unchanged.
- **RAG Is Not a Database**: It follows a “retrieval + generation” paradigm and uses a database as its underlying layer.
- **RAG Is Not a Search Engine**: Its goal is to generate natural language answers, not to return a list of links.
- **RAG Is Not Always More Suitable Than Fine-tuning**: Fine-tuning is still needed to teach the model new abilities or styles.
- **A Reranker Is Not Always Necessary**: While it’s highly recommended, it significantly improves performance.
- **Larger Chunk Sizes Are Not Always Better**: Larger chunks can dilute relevance and waste tokens.

## Case Study 12: "Ask AI About This Article" on This Blog

Taking this blog as an example, the actual architecture is as follows:

```markdown
Markdown article (knowledge base)
   ↓ Automatically segmented (by paragraphs/headers)
   ↓ bge-m3 generates embeddings
   ↓ Stored in a vector database (this blog uses Postgres + pgvector; you can also use Qdrant)
   ↓
User asks a question
   ↓ bge-m3 vectorizes the question
   ↓ The vector database retrieves relevant article segments (Top-15)
   ↓ bge-reranker-v2-m3 reorders the results (Top-5)
   ↓ Calls the free DeepSeek model (or a cheaper alternative) to generate an answer

A complete question-and-answer process (in pseudocode, ready to implement):

// ① Indexing phase (runs once when the article is written)
const chunks = splitByHeading(markdownText);          // Automatically segment the text
const vectors = await embed(chunks);                  // bge-m3 generates embeddings
await db.insert(chunks.map((c, i) => ({ content: c, embedding: vectors[i] }));

// ② Question phase
const queryVec = (await embed([question]))[0];        // Vectorizes the question
const candidates = await db.search(queryVec, 15);     // The vector database retrieves the Top-15 relevant articles
const ranked = await rerank(query, candidates, 5); // bge-reranker reorders the results to get the Top-5
const context = ranked.map(r => candidates[r.index]).join('\n\n');

// ③ Answer generation (only sends 5 relevant segments, which is both cost-effective and accurate)
const answer = await chatWithLLM(`
Please answer based on the following information, and mention the source:
${context}

Question: ${question}
`);

The embedding and reordering processes use the free DeepSeek model:

const SF_KEY = process.env.SILICONFLOW_API_KEY!;

// Embedding: BAAI/bge-m3 (free, 1024 dimensions)
async function embed(texts: string[]): Promise<number[][]> {
  const r = await fetch('https://api.siliconflow.cn/v1/embeddings', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${SF_KEY}` },
    body: JSON.stringify({ model: 'BAAI/bge-m3', input: texts, encoding_format: 'float' }),
  });
  const j = await r.json();
  return j.data.map((d: any) => d.embedding as number[]);
}

// Reordering: BAAI/bge-reranker-v2-m3 (free)
async function rerank(query: string, documents: string[], topN = 5) {
  const r = await fetch('https://api.siliconflow.cn/v1/rerank', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${SF_KEY}` },
    body: JSON.stringify({ model: 'BAAI/bge-reranker-v2-m3', query, documents, top_n: topN }),
  });
  return (await r.json()).results; // Returns the results sorted by relevance score ([index, relevance_score])
}

For the complete SQL code for setting up the pgvector database and the cost comparison, see another of my articles. The main conclusion is: Both the embedding and reordering processes are free, and the generation model only requires about 3000 tokens, which saves more than 99% compared to using a full prompt.

---

Summary

The essence of RAG (Retrieval with Augmentation) can be summed up in one sentence: Store knowledge in a vector database, and when a question is asked, only the most relevant segments are sent to the large model.

• Embedding is done using BAAI/bge-m3 (for free).
• Reordering is done using BAAI/bge-reranker-v2-m3 (for free).
• The vector database can be implemented using pgvector or Qdrant.
• The generation model only considers the Top-5 results.

By following this architecture, your AI-based question-and-answer system will be more accurate, faster, and significantly more cost-effective. If you want to use these two free models for free, register at the following link: https://cloud.siliconflow.cn/i/MBwLMSrY

The code from this article can be used directly; in fact, the “Ask AI About This Article” feature on this blog itself uses the same process.