RIP Dad. RIP Mamaw Joyce. Lawyer. General Counsel, Contributor, Board and Member of Local Inference Lab, a 501(c)(3). AI dork and quanter. lover of puns.
Grayson, KY
- Tweets 43,539
- Following 347
- Followers 2,193
- Likes 19,701
Brandon Music retweeted
Oh hey would you look at that? A new inference engine dropped called Flare.
Optimized specifically for sm12x hardware.
Better performance. More room for KV cache.
All with best in class support for the latest models.
Crazy stuff is happening at @YourLocalAILab and @NVIDIAAI.
2
5
28
8,384
We have been hard work in the kitchen. Proud of the crew @YourLocalAILab .
2
4
13
2,676
Brandon Music retweeted
🚨 @NVIDIAAI just quietly dropped its own local inference server...
No announcement. Single-digit stars
It's called Flare and Apache 2.0
Built for SM120/SM121 GPUs RTX PRO 6000 Blackwell and DGX Spark
>PyTorch + FlashInfer at build time
>Rust runtime at serve time. No Python.
>CUDA graphs captured before serving
>FP8 KV cache by default
>Recipes for Qwen Flash, GLM and DeepSeek
github.com/NVIDIA/flare-loca…
45
86
739
67,323
The results for KLD across the various quants with the trellismx quant technique .
DeepSeek-V4.1-Flash-1017 in TrellisMX K2 Mix, laid out for two RTX PRO 6000s: 2-bit experts, with the 629 that matter most raised to 3- and 4-bit by ShapleyMCG allocation, sized to keep ~3M tokens of KV. Engram tables in 4-bit TrellisMX. Mean KLD 0.1306 nats, 8.6% below plain K2. A +5 GB sibling for ~2M KV is up too. Only KLD measured so far; serving is a test image.
The whole build and every KL-divergence measurement ran on an 8x B200 node generously donated by @LambdaAPI. Thank you, Lambda!
huggingface.co/brandonmusic/…
1
9
539
DeepSeek-V4.1-Flash-1017 in TrellisMX K2 Mix, laid out for two RTX PRO 6000s: 2-bit experts, with the 629 that matter most raised to 3- and 4-bit by ShapleyMCG allocation, sized to keep ~3M tokens of KV. Engram tables in 4-bit TrellisMX. Mean KLD 0.1306 nats, 8.6% below plain K2. A +5 GB sibling for ~2M KV is up too. Only KLD measured so far; serving is a test image.
The whole build and every KL-divergence measurement ran on an 8x B200 node generously donated by @LambdaAPI. Thank you, Lambda!
huggingface.co/brandonmusic/…
3
16
1,723
New: DeepSeek-V4.1-Flash-1017 in 3-bit TrellisMX, laid out for three RTX PRO 6000s, Engram tables at fp8 ngram. Mean KLD 0.0664 nats. With sufficient ram, it's been capable of running two sparks as well.
The whole build and every KL-divergence measurement ran on an 8x B200 node generously donated by @LambdaAPI. Thank you, Lambda!
huggingface.co/brandonmusic/…
2
6
26
4,888
Brandon Music retweeted
It's going to be about a lot more than recipes really soon.
Stay tuned. We're cooking hard. 👨🍳
1
3
20
8,141
Looks like you can get about 4 million kv cache when pinning it with a env flag. C1 decode measured at 120 t/s on tp2. Reasoning tests passed too. Repo is updated.
New: DeepSeek-V4.1-Flash-1017 in 2-bit TrellisMX, laid out for two RTX PRO 6000s, Engram tables at 4-bit in system RAM of 128 gb.
The whole build and every KL-divergence measurement ran on an 8x B200 node generously donated by @LambdaAPI. Thank you, Lambda!
huggingface.co/brandonmusic/…
1
1
6
427
20? I have well over 400 lol.
Microsoft on Thursday launched Copilot for Legal, offering more than 20 legal-specific skills in Microsoft Copilot, bringing contract review, drafting, obligations tracking and other legal workflows into Word and Cowork, its agentic workspace, along with plugins from more than two dozen legal technology partners.
Microsoft now becomes the fourth major AI company in five months to roll out a dedicated legal offering, following Anthropic’s Claude for Legal in May, Google’s Gemini Enterprise for Legal in August, and OpenAI’s Astra for Law last month. - @bobambrogi
lawnext.com/2026/10/microsof…
6
443
And if you want to measure KLD with any deepseek 4.1 1017 quant the teacher logits will be released as well.
New: DeepSeek-V4.1-Flash-1017 in 2-bit TrellisMX, laid out for two RTX PRO 6000s, Engram tables at 4-bit in system RAM of 128 gb.
The whole build and every KL-divergence measurement ran on an 8x B200 node generously donated by @LambdaAPI. Thank you, Lambda!
huggingface.co/brandonmusic/…
1
11
545
New: DeepSeek-V4.1-Flash-1017 in 2-bit TrellisMX, laid out for two RTX PRO 6000s, Engram tables at 4-bit in system RAM of 128 gb.
The whole build and every KL-divergence measurement ran on an 8x B200 node generously donated by @LambdaAPI. Thank you, Lambda!
huggingface.co/brandonmusic/…
9
3
31
2,580
Just like everyone predicted. Kentucky is 6-1 and 4-1 in the SEC, tied with Georgia for the league lead, and Vanderbilt coming to town next week. Defense got it done tonight. Will Stein is the SEC’s coach of the year at midseason point. Three straight SEC road wins for the Cats.
17
54
1,069
84,407
C1 speed is not the issue . It’s about having an agentic workflow where multi agent review, swarm etc can help you build a record, conduct research etc. it’s about having to put all that intelligence to work in real time with large context. It’s why flash models are important.
In the world of the law, how many tasks can you list where immediate response of an AI model is critical? I handled civil and criminal cases for years, I never once remember something like, "wow, I just received a motion from the opposition [for some reason] I have to respond in 20 minutes."
My point is that Legal AI models promising faster answers don't make sense to me. The marketing should be something highlighting less review time on their output, not faster output.
1
1
5
661
Brandon Music retweeted
Get up to 30% more KV cache using our lossless CSF version of @deepseek_ai V4.1-Flash-1017.
No drop in quality, just more room for you to do what you want to do!
Find the weights on our @huggingface page:
huggingface.co/local-inferen…
9
12
59
16,252
shoutout to @YourLocalAILab for pushing forward and doing the work on QAD and lossless csv compression.
Nvidia is actually giving a heck about RTX Pro 6000s and DGX Sparks now.
New FP4 “lossless” quantisation schema, about 8% smaller while maintaining FP8 performance
What’s special is that they specifically published this w DGX Sparks and 6000 recipes w reporting!
3
1
19
1,194
experimenting with Deepseek v4.1flash 1017 (lossless CSF compression) on dcp4 and i get 33,346,277 tokens of KV capacity. at the cost of slower prefill. Without it around 13 million. For right now, i'll not do dcp4 lol.
4
19
972