Our Specialized Sovereign Large Language Model for Mission Critical Environments is here: aleph-alpha.com/en/kolibri/
Heidelberg, Germany
- Tweets 616
- Following 24
- Followers 25,492
- Likes 694
Pinned Tweet
Small bird, fast wings, Kolibri is here.
78B parameters. 3.46B active. Up to 1M tokens of context. Built in Europe.
Now the weights are yours. Run it on your own hardware, under Apache 2.0.
436
1,016
9,834
2,683,254
The location of one space in a multiple-choice evaluation changes what it measures. It changes the probability of the correct answer by several orders of magnitude but usually barely changes the accuracy.
1
2
22
1,397
Given a prompt that ends with a space, we find that Olmo 3 still generates an answer with a space and is semantically correct 90% of the time.
1
2
266
Our recommendation is to split the prompt and the answer so that the tokens match what the model is trained with. This can be checked before any inference, with just a tokenizer: tokenizing the prompt and answer separately should give the same tokens as tokenizing them together. Read the full blog here: aleph-alpha.com/en/blog/mind…
3
276
Expert load balancing is key to training Kolibri efficiently at scale.
We developed two new methods: Exact Quantile Balancing (EQB) for global balance and Load-Error Injection (LEI) for local balance. @JesseLiJiaze and @PitNeitemeier explain them in a new, visual blog post.
1
11
43
4,892
Blog post: aleph-alpha.com/en/blog/bala…
Paper: arxiv.org/abs/2609.28053
4
688
Same model, same question, same meaning, but a longer answer can improve the score ~2x. A large share of bits-per-byte (BPB), a common pre-training eval metric, reflects answer length rather than predictive quality. We propose an alternative, PriorBPB. 🧵
2
6
73
4,985
Across 62 base checkpoints from 5 model families, PriorBPB reduces mean |correlation| with answer length from 0.68 to 0.18. It also tracks accuracy along capability ladders better than mean-of-ratios BPB on both base and instruct checkpoints.
1
7
459
tl;dr: We suggest to not use mean-of-ratios or corpus BPB alone and use PriorBPB with a stated shared prior for cross-task comparisons. Check out the blog post, paper, and Apache 2.0 code by @tfburns and the team here: aleph-alpha.com/en/blog/prio…
8
428