Recently, I decided to take a break from work and spend around 14 days away from the keyboard and the tech and AI world, which I clearly couldn’t do.
I ended up spending the last four days training an AI model from scratch.
The truth is, I treat AI model training more as a hobby that I really enjoy than as something purely commercial. I’m also very interested in the academic side of it and in the open source community.
So I spent my free time over the past few days training a small 0.2B model from scratch on 6 billion training tokens.
The nice part is that I trained the model only on Arabic training data, to specialize it in Arabic and its dialects.
I stopped at the pretraining stage.
The model was trained for around 30 continuous hours on one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU. The total cost of training the base model during pretraining, where I stopped, was 100.63 dollars.
This part is probably going to upset people who think training AI models always costs millions of dollars.
For me, it was just something fun to do in my free time, instead of completely losing my mind, walking around the walls, or watching TV.
Today, I open sourced the model, along with the training data sources, the training code, and everything else, under the MIT License for academic and educational purposes.
Also, I honestly had no idea what to do with the model after training it haha.
And that’s how I spent my free time.
The model is called Horus Taleeq 0.2B Base. Search for it and you’ll find its page on TokenAI.
I hope it can be useful to you.
[link] [comments]