RAG Knowledge Engine

What You Will Build

A full retrieval pipeline that turns raw documents — PDFs, DOCX files, Markdown, even live URLs — into a searchable knowledge base your agent can cite. By the end of this phase, your Phase 2 agent gains a fifth tool, search_knowledge_base, backed by PostgreSQL and pgvector. The agent now answers questions grounded in documents it has actually read, instead of whatever Llama 3.3 memorized during training.

Data Flow Visualization

Watch a document travel through loading, chunking, embedding, and storage — then watch a live query take the mirror path: embed, search, inject, answer. Click any node for the real payload shape at that point in the pipeline.

Why This Phase Exists

Ask Llama 3.3 (or GPT-4o, or Claude) a question about your company's refund policy, your own product's pricing tiers, or an event from last week. It will answer confidently, and it will be wrong. Not because the model is bad, but because the answer was never in its training data. A cleverer system prompt cannot fix this. The model is missing the knowledge.

There are two ways to give a model new knowledge, and they solve different problems:

• ▸Fine-tuning re-trains the model's weights on examples of the behavior you want, such as tone, output format, or a specific reasoning style. It is slow to iterate, because every update means a new training run. It is also expensive at any real scale.
• ▸RAG (Retrieval-Augmented Generation) leaves the model's weights untouched. Instead, at query time, you fetch the few documents most relevant to the question and paste them straight into the prompt as context. The model still generates the answer, but now it reads from your data instead of guessing.

Here is the rule of thumb that settles almost every "should I fine-tune or RAG this?" argument: if the answer already exists inside a document, use RAG. If it is a learned skill or style, fine-tune. Company docs, FAQs, product manuals, anything that changes weekly — use RAG every time, because updating a RAG index is "add a row," not "retrain a model." This phase turns your agent from "knows whatever Llama was trained on" into "knows what your company actually knows." It is the first feature every enterprise pilot asks for.

How the RAG Pipeline Actually Works

RAG is five mechanical steps. Treating them as a checklist rather than a black box is what makes the rest of this phase clear:

1. ▸INGEST — load a document from disk or a URL and pull out raw text (PDF via pdf-parse, DOCX via mammoth, HTML via cheerio).
2. ▸CHUNK — split that text into pieces small enough to embed with clear meaning, and small enough to fit several of them in a single prompt.
3. ▸EMBED — convert each chunk into a 1536-number vector using text-embedding-3-small. The vector is a numeric fingerprint of the chunk's meaning, not just its words.
4. ▸STORE — save each chunk's text next to its vector in document_chunks, a plain Postgres table with a vector(1536) column, provided by the pgvector extension.
5. ▸RETRIEVE — when a user asks a question, embed the question the same way, find the stored chunks whose vectors are closest to it, and inject their text into the LLM's context window before it answers.

Gotcha: steps 1 to 4 (ingestion) run once per document, offline, whenever someone uploads something new. Step 5 (retrieval) runs on every single user message, in the request's hot path. If your chunking or embedding step is slow, that only delays an upload. If your retrieval query is slow, every user of your product feels it on every question. This is why the HNSW index later in this phase is not optional.

Why pgvector for This Phase

The obvious move is to reach for a dedicated vector database, such as Pinecone, Weaviate, or Qdrant. pgvector is the less obvious choice on purpose, and it is the right one here:

• ▸You are already running PostgreSQL since Phase 2 (session storage, and soon the rest of your app's relational data). pgvector is a Postgres extension — CREATE EXTENSION IF NOT EXISTS vector; — not a new product category, new SDK, or new billing account. Phase 3 still swaps the Compose image to pgvector/pgvector:pg16 (same Postgres product, pgvector-enabled build) so the extension and init.sql schema exist on day one.
• ▸Vectors and their metadata live in the same database, in the same transaction, as everything else. You can JOIN document_chunks against documents, against users, against permissions. A dedicated vector store forces you to stitch that join back together across two systems by hand.
• ▸For the corpus sizes real internal knowledge bases actually have (thousands to low millions of chunks, not billions), pgvector's HNSW index is competitive with purpose-built vector databases on latency, at zero extra infrastructure cost.

Gotcha: pgvector is not always the right call. The architecture decision section below shows exactly where it stops being the best option.

Key RAG Parameters

Transactional Ingestion You Cannot Skip

Problem: ingesting a document means writing to two tables. First one row in documents, then one row per chunk in document_chunks (often dozens). If your process crashes, or an embedding call throws, halfway through inserting chunk 30 of 80, you are left with a documents row that claims a document exists with only a third of its content searchable. And no error tells you that.

vectorstore.service.ts wraps the whole ingest in a single Postgres transaction:

 Copy
await client.query('BEGIN');
// ...insert into documents, insert every chunk...
await client.query('COMMIT');
// on any thrown error:
await client.query('ROLLBACK');

Either every chunk for a document lands, or none do. There is no partially-ingested document that a retrieval query could quietly return incomplete results from.

The second half of this problem is the upload itself, before ingestion even starts. multer({ dest: '/tmp/', limits: { fileSize: 10 * 1024 * 1024 } }) caps uploads at 10MB for the same reason Phase 1 capped JSON body size. An upload endpoint with no limit is a way to fill your disk and run up embedding cost, on a route that anyone who finds it can reach.

Embedding Dimension — Fail-Fast Pattern

The document_chunks.embedding column is declared as vector(1536). This is a fixed width, baked into the schema at CREATE TABLE time. text-embedding-3-small outputs exactly 1536 numbers, so today everything lines up.

Gotcha: nothing in the application code enforces that match. Suppose a future EMBEDDING_MODEL swap points at a model with a different output size, say a 1024-dimension model. The failure does not happen at deploy time, and it does not happen at startup. It happens the moment the first document is ingested, as a raw Postgres error (expected 1536 dimensions, not 1024) thrown from deep inside an INSERT. This is the same "fail loud, fail early" lesson from Phase 1's env validation, just moved to a new spot. The fix is to validate the embedding model's declared dimension against the schema once, at startup, next to your other env checks. Do not discover the mismatch from a stack trace during a live ingest.

Core RAG Pipeline — Code Deep Dive

The two functions that do all the real work live in vectorstore.service.ts. Ingestion chunks, embeds, and stores, all in one transaction:

 Copy
async ingestDocument(name: string, content: string, source: string, type: string) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const { rows } = await client.query(
      'INSERT INTO documents (name,source,type) VALUES ($1,$2,$3) RETURNING id',
      [name, source, type]
    );
    const docId = rows[0].id;
    const chunks = chunkText(content);
    const vectors = await embedChunks(chunks.map(c => c.content));
    for (let i = 0; i < chunks.length; i++) {
      await client.query(
        'INSERT INTO document_chunks (document_id,content,embedding,chunk_index) VALUES ($1,$2,$3::vector,$4)',
        [docId, chunks[i].content, '[' + vectors[i].join(',') + ']', i]
      );
    }
    await client.query('COMMIT');
    return { documentId: docId, name, chunksCreated: chunks.length };
  } catch (e) { await client.query('ROLLBACK'); throw e; }
  finally { client.release(); }
}

Retrieval embeds the query, then lets Postgres do the nearest-neighbor math with the <=> cosine-distance operator that pgvector adds:

 Copy
async similaritySearch(query: string, topK = 5, threshold = 0.65) {
  const vec = '[' + (await embedQuery(query)).join(',') + ']';
  const { rows } = await pool.query(
    `SELECT dc.content, d.name doc_name, dc.chunk_index,
     1-(dc.embedding<=>$1::vector) similarity
     FROM document_chunks dc JOIN documents d ON d.id=dc.document_id
     WHERE 1-(dc.embedding<=>$1::vector)>$3
     ORDER BY dc.embedding<=>$1::vector LIMIT $2`,
    [vec, topK, threshold]
  );
  return rows;
}

Let's walk through it. dc.embedding<=>$1::vector is pgvector's cosine-distance operator. Smaller means more similar, which is why 1 - distance turns it back into an intuitive "similarity" score. The WHERE clause is the threshold from the table above doing its job: it drops weak matches before they reach the LLM, not after. And none of this vector math happens in your Node process. It happens inside Postgres, next to the index. That is what makes the HNSW index below matter.

Remember: this similaritySearch path is the pure-vector helper. The HTTP search route and the agent tool use hybridSearch (vector + BM25 fused with RRF), which does not apply the 0.65 threshold.

pgvector vs Pinecone vs Elasticsearch

This is the second interview question this phase prepares you to answer with confidence: "why not just use a managed vector database?"

• ▸Elasticsearch / OpenSearch — mature full-text (BM25) search, with vector search added later. A strong choice if keyword search is your primary need and vectors are secondary. It is heavier to operate than Postgres for a team that does not already run it.
• ▸Pinecone (or Weaviate, Qdrant) — purpose-built vector databases. They are the better choice once you are at hundreds of millions of vectors and need multi-region replication and vendor-managed scaling. Most products do not start at that scale or with that ops budget.
• ▸pgvector — the right choice whenever your vectors are a feature of an app that already has a relational database, not the entire product. You get transactional consistency between vectors and the rest of your data, one system to back up and operate, and, through the hybrid search below, direct access to Postgres's own full-text search in the same query.

Gotcha: this decision is not permanent. If your knowledge base grows into the hundreds of millions of chunks with multi-region latency needs, pgvector's single-primary Postgres model becomes the bottleneck, and moving to a dedicated vector store becomes the right call. But that is a scale problem to solve when you actually hit it, not one to pre-optimize for on day one.

HNSW Indexing — Why Now, Not Later

Problem: similaritySearch above compares the query vector against every row in document_chunks unless an index tells Postgres a smarter way to search. That is an O(n) scan. It is fine at 500 chunks and unusable at 500,000.

 Copy
CREATE INDEX IF NOT EXISTS chunks_embedding_hnsw
ON document_chunks USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

HNSW (Hierarchical Navigable Small World) builds a multi-layer graph over your vectors when the index is created. A search at query time can then skip through layers instead of checking every row, which turns that O(n) scan into an approximate O(log n) lookup. "Approximate" is the honest word: HNSW can sometimes miss the single mathematically-closest chunk in exchange for being much faster on every query. Every production RAG system makes this trade on purpose.

Gotcha: creating this index costs real time and memory up front, in proportion to how many chunks already exist. So it is cheap and painless on an empty table on day one, and increasingly expensive to build for the first time against a table you let grow unindexed for months. Add the index in the same migration that creates the table, not as a "we'll optimize search later" follow-up.

Test It — curl Commands + Expected Output

This is the verification gate. Run one phase stack at a time. Volumes do not collide (agenttelar-phase3 vs agenttelar-phase2), but host ports do: Phase 2 and Phase 3 both publish :3000, :6379, and :5432. From your phase3/ folder:

 Copy
# Required if Phase 2 is still running — frees shared host ports (volumes stay):
(cd ../phase2/docker && docker compose down)

cp apps/api/.env.example apps/api/.env
# Open apps/api/.env and set GROQ_API_KEY and OPENAI_API_KEY
# (embeddings use OpenAI; the chat agent still uses Groq)

cd docker && docker compose up --build -d

Phase 3 uses pgvector/pgvector:pg16 — same Postgres product, pgvector-enabled image. Once docker compose ps shows healthy services, run the checks below in order.

1. Confirm the extension is actually enabled (do this before anything else):

 Copy
docker compose exec postgres psql -U agenttelar -c \
  "SELECT * FROM pg_extension WHERE extname='vector';"
# Should return one row — if empty, init.sql did not run (wipe once: docker compose down -v && up --build -d)

2. Ingest a real refund policy (do not use example.com — it has no refund text):

 Copy
cat > /tmp/Refund-Policy.md <<'EOF'
# Acme Refund Policy

Customers may return unused items within 30 days of purchase.

Approved refunds are processed within **5–7 business days** and credited
to the original payment method. Shipping fees are non-refundable unless
the return is due to our error.
EOF

curl -X POST http://localhost:3000/api/rag/ingest/file \
  -F 'file=@/tmp/Refund-Policy.md'
# {"documentId":"...","name":"Refund-Policy.md","chunksCreated":N}  with N > 0

3. Search it directly:

 Copy
curl "http://localhost:3000/api/rag/search?q=how+long+do+refunds+take"
# {"query":"how long do refunds take","results":[
#   {"content":"...5–7 business days...","doc_name":"Refund-Policy.md","chunk_index":0,"score":...}
# ]}

4. Ask the agent — it should reach for the new tool on its own:

 Copy
curl -X POST http://localhost:3000/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"message": "How long do refunds take according to our policy?"}' \
  --no-buffer -i
# Copy sessionId from the X-Session-ID response header (also present on each SSE event):
SESSION_ID=<paste-from-header>

curl http://localhost:3000/api/chat/session/$SESSION_ID/history
# toolsUsed should include "search_knowledge_base"

Expected: the agent calls search_knowledge_base and the answer mentions 5–7 business days. The exact citation wording varies — filename optional — as long as it is grounded in the ingested policy, not a generic ungrounded reply. If it answers without using the tool, registration is the first thing to check.

Try It Live

Run the actual phase_3 code in your browser. No local install needed. Once the editor loads, copy .env.example to .env and set GROQ_API_KEY plus OPENAI_API_KEY. Full RAG + session memory also needs Postgres with pgvector and Redis (DATABASE_URL + REDIS_URL); hosted setups are possible but awkward. Full RAG + memory is easiest on the local Docker path above.

Cost Awareness — Numbers Table

Embeddings are billed separately from the LLM call that generates the final answer, and they are far cheaper. But they scale with ingestion volume, not just query volume. This surprises people the first time they bulk-upload a document set.

Completion Checklist — Technical

Before you consider Phase 3 done, all of these should be true:

• ▸SELECT * FROM pg_extension WHERE extname='vector' returns a row
• ▸POST /api/rag/ingest/file returns { chunksCreated: N } with N > 0
• ▸GET /api/rag/search?q=... returns ranked results with score values
• ▸The agent invokes search_knowledge_base when asked about ingested content — verified by checking toolsUsed in the session history (GET /api/chat/session/:id/history)
• ▸The agent's answer mentions 5–7 business days from the ingested policy

Optional stretch (inspect the code, or try once — not required for the quiz):

• ▸EXPLAIN ANALYZE on a cosine-distance query shows an Index Scan using chunks_embedding_hnsw, not a Seq Scan
• ▸Uploading a file over 10MB is rejected by multer (error / reject — status may be 413 or 500 depending on Express error handling), not silently accepted
• ▸A deliberately-failed ingest (kill the process mid-insert) leaves zero orphaned chunk rows — confirming the transaction actually rolled back

Conceptual Checklist — Can You Explain?

If you can't answer these out loud without checking your code, re-read the relevant section above before moving on:

• ▸Why does chunk size affect retrieval quality more than which embedding model you pick?
• ▸What is cosine similarity, and why is it preferred over Euclidean distance for text embeddings?
• ▸What is HNSW, and why does it turn an O(n) search into an approximate O(log n) one?
• ▸What does Reciprocal Rank Fusion do, and how does it combine a vector ranking with a keyword (BM25) ranking?
• ▸When would RAG be the wrong tool, and fine-tuning the right one?
• ▸Why is the entire ingest wrapped in a single database transaction?
• ▸Why is pgvector often the better default over Pinecone — and when does that stop being true?
• ▸What breaks, and where, if the embedding model's output dimension no longer matches the vector(1536) column?

What You Will Learn

• ▸Build a complete RAG pipeline: ingest, chunk, embed, store, retrieve
• ▸Understand embeddings, cosine similarity, and the HNSW index deeply, not superficially
• ▸Implement hybrid search — vector similarity fused with keyword (BM25) ranking via Reciprocal Rank Fusion
• ▸Give your Phase 2 agent a real, citable company knowledge base

Next Steps

Clone the phase3 folder on GitHub. Stop Phase 2 with docker compose down in phase2/docker (frees :3000, :6379, :5432), copy apps/api/.env.example to apps/api/.env, set both GROQ_API_KEY and OPENAI_API_KEY, then cd docker && docker compose up --build -d. Work through the curl tests above until each one matches. Complete the quiz before Phase 4.