Continuing our theme of running large language models (LLM) locally on your PC for freeⁱ; in this post we build upon the local AI chatbot from part 1, improving the responses by retrieving data to make it more knowledgeable. To do this, we use a technique called retrieval augmented generation (RAG), that adds data to the prompt before it goes to the model.

Conventional wisdom says RAG is a magic bullet that stops LLMs from going off topic, and eliminates hallucinations. But, is this really true?

Let’s explore further by diving into how RAG works, and run a simple demonstration.