Agaram 2.5B update 🧵 Yes, it went from 1.5B to 2.5B - part of the reason I haven’t made updates was because when 1.5b version was done, it was not good enough. It could not stop and was rambling, and was looping responses. After that, I went into a full on rework without any idea if it will work. Studied a lot more papers than I want to admit, and revised the corpus and added twice the amount of synthetic data.
At one point, I was close to stopping it and releasing a model with a write-up of where I failed. However, I guess I am just too stubborn- so blew another big chunk of money and time and decided to do it all again the right way. So after another two months - it is almost done.
A first-of-its-size, Tamil-first language model trained from scratch at Murai Labs: ~50B tokens, its own tokenizer, Tamil + Tanglish chat, tool calling.
Pretraining ✅ Chat fine-tuning ✅
Training a final round now on identity, tool use and safety.
Next week: results, and open weights if they hold up. அகரம் வருகிறது 🌱