Nanbeige LLM Lab, BOSS Zhipin (Kanzhun Limited)
Beijing
- Tweets 74
- Following 443
- Followers 5,771
- Likes 148
🚀Excited to announce Nanbeige4.3-33B!
To push the performance ceiling of small agents, our strategy in Nanbeige4.3 starts with building a powerful teacher first.
Nanbeige4.3-33B delivers SOTA agentic performance among open models at ~30B parameters:
* Strong results on agentic coding and cowork benchmarks.
* When integrated into mainstream agent harnesses, it tackles real-world tasks with cost-efficient inference.
* It will serve as a teacher for upcoming smaller Nanbeige models, starting with Nanbeige4.3-3B.
huggingface.co/Nanbeige/Nanb…
huggingface.co/Nanbeige/Nanb…
#LLM #Agent
246
368
3,120
562,935
Thanks for @ArtificialAnlys 's sharing. Please stay tuned for our upcoming 3B model! For those who need quantization, we recommend using our official quantized versions (huggingface.co/collections/N…). Thank you!
Nanbeige has released Nanbeige4.3-33B, an open weights model matching Qwen3.8 27B (xhigh) at 20 on the Artificial Analysis Intelligence Index v5 with stronger terminal use but weaker knowledge
@Nanbeige is the foundation model research team at BOSS Zhipin, the Chinese recruitment platform operated by Kanzhun Limited. Its dense 33B reasoning model is open weights under Apache 2.0, with text input and output only, a 524K-token context window, and is small enough to run locally on a laptop.
Key results:
➤ Competitive results at ~30B total parameters. Nanbeige4.3-33B and Qwen3.8 27B (xhigh) both score 20, compared with 6 for Muse Glimmer (high, 30B).
➤ Terminal use and physics reasoning are relative strengths. Nanbeige scores 30% on Terminal-Bench 4.0 and 16% on CritPt v2's Hard Subset, vs. 6% and 4% respectively for Qwen3.8 27B.
➤ Different strengths to Qwen3.8 27B despite the same overall score. Nanbeige beats Qwen3.8 27B on AutomationBench-AA (53% vs. 48%), but trails on AA-Briefcase v1.1 (1162 vs. 1391 Elo), FrontierCode 2.0 (11% vs. 15%) and HLE-Diamond (7% vs. 15%).
➤ High abstention and low hallucination rates on Omniscience. Nanbeige scores -4 on AA-Omniscience v1.1 vs. -6 for Qwen3.8 27B, with a hallucination rate of 18% vs. 29%. It answers fewer questions correctly (12% vs. 17%) and declines 71% of questions.
➤ Similar intelligence with more output tokens per task. Nanbeige used 158K output tokens per Intelligence Index task vs. 110K for Qwen3.8 27B, about 44% more.
Additional model details:
➤ Type: Open weights reasoning (Apache 2.0 license)
➤ Size: 33B total parameters (dense)
➤ Modalities: Text input and output only
➤ Context window: 524K
➤ API availability: No confirmed public first- or third-party APIs
8
2
124
3,813
We introduce Depth Hybrid Attention (DHA), which incorporates depth-wise value mixing into hybrid architectures with very little additional compute and cache overhead. By allowing layers to mix information across depth, DHA broadens each layer’s view and increases the model’s expressive capacity.
Our DHA method adapts to multiple hybrid architectures, including Depth Window Attention (DWA) for GQA + SWA and Depth Delta Attention (DDA) for MLA + KDA.
Across loss curves, short-context benchmark results, and long-context retrieval evaluations, both variants outperform their corresponding full-attention and hybrid-attention baselines, while retaining hybrid attention’s low KV-cache footprint.
Project overview & code: github.com/Nanbeige/Depth-Hy…
14
49
343
23,926
With the release of OpenAI's GPT-6 Astra @openai and @rasbt's widely shared article, Nanbeige4.2-3B ( huggingface.co/Nanbeige/Nanb…) is back in the spotlight! 🚀
Built on a Looped Transformer, our 3B model consistently outperforms larger models (e.g., Qwen3.5-9B, Gemma4-12B) across key Agent benchmarks: SWE-Bench Pro, Terminal-Bench, GDPval, and more.
Nanbeige4.5 is currently training with LoopSplit, mHC + depth attention, and concatenated n-gram embeddings. Stay tuned!
A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
Also, the statement "the new technique works in a way that obscures some or all of the AI's reasoning, otherwise known as 'chain-of-thought'" is not necessarily true with respect to the looped transformer method. It's possible that The Information journalist refers to some other technique or misunderstood the looped transformer method.
Reusing layers does not by itself suppress visible chain of thought. It adds computation in hidden states before the next token is emitted, just as ordinary transformer layers do.
But based on the information we have, the only plausible interpretation here is that if a model uses more of these recurrent passes, it may need to generate fewer intermediate reasoning tokens. So then more of its computation happens in latent activations that cannot be read as text. But we would get the same effect if we were scaling up the model size, like GPT 5.6 Luna -> GPT 5.6 Sol.
16
46
395
28,594
We just released the DSpark weights for Nanbeige4.2-3B (huggingface.co/Nanbeige/Nanb…).
Osaurus is coming to Windows.
Same idea. Your agents, your files, your models, on your machine.
This is a 3B model reading my spreadsheets and finding a margin problem I missed.
4
12
92
10,887
Artificial Analysis @ArtificialAnlys just dropped their new mobile device benchmarks, and Nanbeige 4.2-3B ranks TOP 1 in Average Score (16K max context).
At the standard 16K context limit, Nanbeige4.2-3B (Reasoning) and LFM2.5-2.6B (Reasoning) tie for the top average evaluation score at 63, ahead of Ornith-1.0-9B (Reasoning, 62), Qwen3.5 9B (Reasoning, 61), Ornith-1.5-9B (Reasoning, 61), Gemma 4 E4B (Reasoning, 60) and Qwen3.5 9B (Non-reasoning, 60). Ornith-1.5-9B slips behind its older 1.0 sibling purely on a weaker instruction following performance in IFBench
When the context limit is raised to 64K (represented by dots in the image), Ling 3.0 Tiny takes the top spot at 66, followed by Nanbeige4.2-3B at 65, and both Qwen3.5 9B (Reasoning) at 64 and LFM2.5-2.6B at 64
13
14
246
22,689
Nanbeige retweeted
Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up outside the closed labs. Plus, it gives us the freedom to run AI on our own hardware if we are not ready to share personal data and IPs with closed labs through using their models. (Not that proprietary models are bad, actually I use them a lot as well, but it wouldn't healthy not to have any alternatives.)
Anyway, while pretty much everyone is waiting for the Kimi K3 and Ling 3.0 weights to land on the model hub any day now, there were quite a few other interesting new open-weight model releases the past week. Yes, one of those weeks!
So, here are the architecture pics along with some notes on what I found most interesting:
1) Nanbeige 4.2 3B uses looped depth sharing. This basically means it runs the same 22-layer (=transformer block) stack twice. So, it extends the 22-layer architecture to 44-layers, but without duplicating the weights. (2x the transformer block compute but same memory footprint.)
Why? The info is a bit sparse, but section 2.1 of the Nanbeige 4.2 technical report says two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. More passes gave barely any gains but made the training much slower and much more expensive.
2) Laguna S 2.1 is poolside's Laguna model in a really nice size: 118B sparse MoE with 8B active parameters and a 1M-token context window. Otherwise, the architecture is pretty standard. It uses 36 sliding-window and 12 global (gated-)GQA layers. However, given this size, and the fact that it (just barely) runs on my DGX Spark (uses about <80 GB of RAM), this is right now the most interesting model for me personally. It's 3x bigger and thus a tad slower but maybe a good candidate as daily-driver-Qwen3.6-35B-replacement. (Still waiting on some more independent performance benchmarks though.)
3) Motif-3-Beta is a new 314B-A13B sparse MoE that is somewhat based on DeepSeek V4 in terms of mHC and latent attention. But it uses a new component, Grouped Differential Latent Attention, which is inspired by Multi-head Latent Attention. I probably should write an article about this some time, but for now, the tl;dr is as follows. Regular MLA compresses the keys and values into a smaller latent representation to mainly reduce the KV cache size. GDLA does a similar low-rank compression but puts the attention heads into groups and also learns a noise head for each group where the noise gets subtracted for filtering purposes... Anyway, a topic for another day!
4) Solar Open 2 is a new 250B-A15B hybrid MoE by Upstage that interleaves three Kimi Delta Attention layers with one GQA layer.
5) Antares 1B is a small model (and there is also an even smaller 0.3B variant) from Cisco starts that with the IBM Granite 4.0 1B backbone and uses SFT plus GRPO for terminal-based cybersecurity stuff. It is a nice example of task-specific post-training on a genuinely small model.
6) BTL-3 is a rank-32 LoRA adapter for Qwen3.6-27B aimed at coding agents and structured tool use. The really strong benchmark performance suggests that LoRA adapters are still a useful tool/technique in 2026.
I added all six to the LLM Architecture Gallery for some additional details:
sebastianraschka.com/llm-arc…
92
256
1,603
75,396
We’re excited by how far a 3B model can go—and even more excited about what comes next for compact local agents. @openclaw
We released Nanbeige4.2-3B, its Looped Transformer increases model capacity without adding parameters, delivering a capable 3B agent.
Nanbeige4.5 is training with LoopSplit, mHC+depth attention & concatenated n-gram embeddings,already in the modeling code.
huggingface.co/Nanbeige/Nanb…
11
43
429
42,082
Nanbeige4.2-3B technical report is out on @huggingface !
huggingface.co/Nanbeige/Nanb…
We released Nanbeige4.2-3B, its Looped Transformer increases model capacity without adding parameters, delivering a capable 3B agent.
Nanbeige4.5 is training with LoopSplit, mHC+depth attention & concatenated n-gram embeddings,already in the modeling code.
huggingface.co/Nanbeige/Nanb…
6
21
175
40,699
Nanbeige retweeted
A 2.4GB model just went toe-to-toe with today’s best local LLMs… and almost beat a 26B MoE. On an RDNA1 GPU.
Nanbeige4.2-3B on Agon-Bench:
• 84.2% composite
• 44.9 tok/s on an AMD RX 5700 XT
• 2.4GB VRAM
Against the current field:
@deep_reinforce Ornith-1.0-9B (5.3GB, 82.7%)
Nanbeige wins overall 84.2 → 82.7. Agent is a massacre: 95.8% vs 81.9%. Ornith takes reasoning (98.0 vs 90.2). Code is effectively tied (66.7 vs 68.0). A 3B model matching a 9B on code while decisively outperforming it on agent tasks is not something I expected.
@GoogleAI Gemma4-26B-A4B (16.9GB, 84.6%)
This is the crazy one. Nanbeige finishes just 0.4 points behind a model nearly 7× larger in VRAM, while running 3.3× faster. Agent is tied at 95.8%. Nanbeige actually wins code (66.7 vs 60.0). Gemma still owns reasoning (98.0 vs 90.2), but the efficiency gap is staggering.
@PrismML Bonsai-27B-Q1_0 (3.8GB, 82.2%)
Two radically different approaches to “small.” Bonsai compresses a 27B model into 1-bit ternary weights. Nanbeige instead loops its own transformer layers. Nanbeige wins overall (84.2 → 82.2) while running 3.6× faster.
The architectural trick is what makes this interesting.
Nanbeige has only 22 physical transformer layers, but each layer is executed twice, giving 44 effective passes through the network. Instead of adding parameters, it adds depth. That appears to be enough to produce a 95.8% agent score, competing with models five to seven times its size.
The weakness is still knowledge-heavy coding. Regex Engine (33%) and String Cleaner (53%) drag the score down. Layer looping can create more reasoning depth, but it can’t invent training data that isn’t there.
This is one of the most impressive efficiency results I’ve seen on consumer hardware. Small models are catching up much faster than I expected.
Custom GGUF and @Italianclownz ROCmFPX recipe to run on RDNA at home in replies ⬇️
We released Nanbeige4.2-3B, its Looped Transformer increases model capacity without adding parameters, delivering a capable 3B agent.
Nanbeige4.5 is training with LoopSplit, mHC+depth attention & concatenated n-gram embeddings,already in the modeling code.
huggingface.co/Nanbeige/Nanb…
1
2
15
1,654
Nanbeige retweeted
🚀 Nanbeige4.2 @nanbeige lands on ModelScope with two models:
- 28T-token Looped Transformer base
- 3B SFT+RL agent for tools, coding, reasoning, office work & deep research
📊 Team-reported wins over Qwen3.5-9B and Gemma4-12B on agent benchmarks.
🤖 modelscope.ai/collections/na…
14
36
318
20,915
We released Nanbeige4.2-3B, its Looped Transformer increases model capacity without adding parameters, delivering a capable 3B agent.
Nanbeige4.5 is training with LoopSplit, mHC+depth attention & concatenated n-gram embeddings,already in the modeling code.
huggingface.co/Nanbeige/Nanb…
36
124
927
217,336
We've just released our technical report(huggingface.co/Nanbeige/Nanb…), feel free to share.
13
44
343
66,582
Nanbeige retweeted
Congrats! Now everyone with iOS devices can try the Nanbeige4.1-3B model immediately on their phone.
This model excels at tool calling and tends to output many thinking tokens, which requires a large context window. I set 12K context on my iPhone 16 Pro Max with 8K max output, and it didn’t crash. If you have a newer device, you can try higher settings, because this model really thinks a lot. I tested the search_web and fetch_url tools. It’s fully compatible out of the box with existing tool chains.
Try it on Privacy AI. It’s completely free for local models.
P.S. I ran it on a real iPhone and recorded the video via iPhone mirroring, which is why the refresh rate isn’t great.
1
19
4,240
🚀 Announcing Nanbeige4.1-3B – our latest open-source 3B model mastering reasoning, preference alignment, & agent capabilities!
Try it now:
huggingface.co/Nanbeige/Nanb…
teddit.net/r/LocalLLaMA/comm…
43
117
855
197,173
Nanbeige retweeted
Intriguing new model called 'Nanbeige/Nanbeige4.1-3B' released, appears to be *extremely* SOTA for its size range. So much so that I question if benchmaxxed. But @nanbeige appears to be a small but real lab out of China so I have faith! Quite exciting - will test.
13
24
248
83,417