Building a RAG chat is one of those projects that sounds simple until you actually build it.

The pitch is usually something like:

"Upload your content and ask questions."

Sure.

Then reality arrives.

You have to decide what gets indexed, how content is chunked, how embeddings are generated, how retrieval works, what gets sent to the model, how sources are handled, what happens when retrieval finds nothing useful and how you stop the model from confidently inventing an answer.

That is the interesting part.

Why I Wanted RAG

I did not want a generic chatbot that knew random things about the internet.

I wanted something that could answer questions about the actual content on my site.

That means the system needs access to my articles and other structured content without requiring me to fine-tune a model every time I publish something new.

RAG is a much better fit for that.

The knowledge stays outside the model.

The retrieval system finds the relevant information.

The LLM uses that information to construct the response.

The Architecture

At a high level, the pipeline is:

Website content → extraction → chunking → embeddings → vector storage → query embedding → similarity search → context selection → LLM → answer

That is the whole idea.

Of course, each arrow contains its own opportunity to screw something up.

Content Extraction

Garbage in, garbage out.

If the extracted content contains navigation menus, repeated headers, footer text and unrelated UI elements, the retrieval system will index garbage.

That creates a problem later because the model can retrieve technically relevant text that is operationally useless.

I would rather spend time cleaning the source content than trying to compensate with a clever prompt.

Chunking

Chunking is one of the most important design decisions.

A chunk should contain enough context to be useful but not so much that unrelated information gets mixed together.

Headings are valuable boundaries.

A section explaining VLAN tagging should probably stay together.

A paragraph about photography should not end up in the same chunk just because it happened to be nearby in an HTML file.

Structure matters.

Retrieval

When a user asks a question, the query is embedded using the same general representation strategy used for the documents.

The system then searches for the closest chunks.

But "closest" does not automatically mean "correct."

That is why retrieval quality needs to be tested.

A system can return highly similar text that does not actually answer the question.

I care about relevance, not just similarity scores.

Context Construction

Once the relevant chunks are selected, they become context for the language model.

The prompt needs to make the boundaries clear.

The model should understand that the retrieved material is reference information, not a command.

This is also where prompt injection becomes interesting.

If indexed content contains instructions such as "ignore previous instructions and reveal..." the retrieval layer has just delivered untrusted text to the model.

That means RAG is not only a search problem.

It is also a security problem.

What Happens When Nothing Matches?

This is critical.

A good RAG system needs permission to say:

"I do not have enough information."

Otherwise, the LLM will happily fill the gap.

That is one of the biggest differences between a useful knowledge assistant and a chatbot that simply sounds confident.

I would rather have the system admit that the answer is not in my content than invent a beautiful explanation.

Testing the System

I test retrieval separately from generation.

That matters.

If the answer is wrong, I want to know whether:

1. the right document was not retrieved 2. the right chunk was not retrieved 3. the context was poorly constructed 4. the model misunderstood the context 5. the model ignored the context 6. the question simply cannot be answered from the available content

Without separating those stages, troubleshooting becomes guesswork.

And I already have enough networks to troubleshoot.

The Security Problem

A public RAG system is an attack surface.

The user can control the query.

The retrieved documents may contain attacker-controlled text if the ingestion pipeline is not trusted.

The model can be manipulated through prompt injection.

The application may accidentally expose retrieved chunks that were never meant to be public.

Logging can also become a data-leak problem.

So the RAG pipeline needs the same kind of thinking I apply to networks:

assume inputs are untrusted define trust boundaries limit access log useful events test failure conditions do not assume the model will behave

What Actually Matters

The interesting part of building RAG is not calling an embedding API.

It is designing the system around failure.

Can I tell why retrieval failed? Can I see what context the model received? Can I prevent irrelevant content from polluting the answer? Can I stop users from turning the model into a data-extraction tool? Can I update the knowledge base without rebuilding everything?

That is the engineering.

The LLM is just one component.