Combining RAG with Continued Pretraining of LLMs

For leaning purposes, I ran an experiment where I used Unsloth to train a model on a new domain using continued pretraining. Then to make it more flexible, I added a RAG step to inject dynamic data to augment the stable training data.

The specific example is training qwen 3.5 4B on a fictional subway system to where the model learns the map well enough to provide travel routes (including multiple transfers, etc). A key point here is to avoid a training corpus that relies on memorization.

Once the subway map was stable and generalized through CPT, I added a RAG step to support dynamic travel announcements (e.g. station closure, concerts near a station, etc).

It's definitely been a fun experiment. Check it out here in case you are interested:

https://www.teachmecoolstuff.com/viewarticle/combining-rag-with-continued-pretraining-of-llms