We are a local AI-focused lab researching and developing optimized performance for vLLM on SM12x hardware for open models.
United States
- Tweets 87
- Following 14
- Followers 2,559
- Likes 216
Pinned Tweet
Get up to 30% more KV cache using our lossless CSF version of @deepseek_ai V4.1-Flash-1017.
No drop in quality, just more room for you to do what you want to do!
Find the weights on our @huggingface page:
huggingface.co/local-inferen…
9
12
60
16,022
We are just getting started.
Oh hey would you look at that? A new inference engine dropped called Flare.
Optimized specifically for sm12x hardware.
Better performance. More room for KV cache.
All with best in class support for the latest models.
Crazy stuff is happening at @YourLocalAILab and @NVIDIAAI.
3
14
3,353
Local Inference Lab 🧪 retweeted
Oh hey would you look at that? A new inference engine dropped called Flare.
Optimized specifically for sm12x hardware.
Better performance. More room for KV cache.
All with best in class support for the latest models.
Crazy stuff is happening at @YourLocalAILab and @NVIDIAAI.
2
5
26
7,376
Local Inference Lab 🧪 retweeted
@YourLocalAILab ftw
🚨 @NVIDIAAI just quietly dropped its own local inference server...
No announcement. Single-digit stars
It's called Flare and Apache 2.0
Built for SM120/SM121 GPUs RTX PRO 6000 Blackwell and DGX Spark
>PyTorch + FlashInfer at build time
>Rust runtime at serve time. No Python.
>CUDA graphs captured before serving
>FP8 KV cache by default
>Recipes for Qwen Flash, GLM and DeepSeek
github.com/NVIDIA/flare-loca…
1
1
10
763
Local Inference Lab 🧪 retweeted
We have been hard work in the kitchen. Proud of the crew @YourLocalAILab .
The crew at @YourLocalAILab along with @NVIDIAAI have been cooking hard. 😎
2
4
11
2,419
Local Inference Lab 🧪 retweeted
The crew at @YourLocalAILab along with @NVIDIAAI have been cooking hard. 😎
🚨 @NVIDIAAI just quietly dropped its own local inference server...
No announcement. Single-digit stars
It's called Flare and Apache 2.0
Built for SM120/SM121 GPUs RTX PRO 6000 Blackwell and DGX Spark
>PyTorch + FlashInfer at build time
>Rust runtime at serve time. No Python.
>CUDA graphs captured before serving
>FP8 KV cache by default
>Recipes for Qwen Flash, GLM and DeepSeek
github.com/NVIDIA/flare-loca…
3
9
7,119
Local Inference Lab 🧪 retweeted
The results for KLD across the various quants with the trellismx quant technique .
DeepSeek-V4.1-Flash-1017 in TrellisMX K2 Mix, laid out for two RTX PRO 6000s: 2-bit experts, with the 629 that matter most raised to 3- and 4-bit by ShapleyMCG allocation, sized to keep ~3M tokens of KV. Engram tables in 4-bit TrellisMX. Mean KLD 0.1306 nats, 8.6% below plain K2. A +5 GB sibling for ~2M KV is up too. Only KLD measured so far; serving is a test image.
The whole build and every KL-divergence measurement ran on an 8x B200 node generously donated by @LambdaAPI. Thank you, Lambda!
huggingface.co/brandonmusic/…
1
8
484
It's going to be about a lot more than recipes really soon.
Stay tuned. We're cooking hard. 👨🍳
1
3
20
7,999
Local Inference Lab 🧪 retweeted
Local Inference Lab is cooking! Get DeepSeek V4.1-Flash-1017 with up to 30% more room for KV cache!
All you need are these drop-in replacement weights and you can have lossless compression.
I use it myself and it makes a big difference. Gives me about 3 million more tokens!
Get up to 30% more KV cache using our lossless CSF version of @deepseek_ai V4.1-Flash-1017.
No drop in quality, just more room for you to do what you want to do!
Find the weights on our @huggingface page:
huggingface.co/local-inferen…
5
5
21
7,589
Fantastic work by @YourLocalAILab !! CSF is 🔥🔥🔥
Our team created the technology behind CSF to give your local machines even more capacity for long workflows, large swarms and other tasks.
It's lossless, too. Meaning you don't lose any of the data you are caching by using it.
Enjoy your new free KV cache tokens!
1
2
5
3,188
Local Inference Lab 🧪 retweeted
Want to run DeepSeek V4.1-Flash-1017 on just two RTX Pro 6000s?
My buddy @BrandonMusicKy has you covered.
This guy is a wizard when it comes to quantizing models.
Recommend giving him a follow and tracking his work.
New: DeepSeek-V4.1-Flash-1017 in 3-bit TrellisMX, laid out for three RTX PRO 6000s, Engram tables at fp8 ngram. Mean KLD 0.0664 nats. With sufficient ram, it's been capable of running two sparks as well.
The whole build and every KL-divergence measurement ran on an 8x B200 node generously donated by @LambdaAPI. Thank you, Lambda!
huggingface.co/brandonmusic/…
2
5
17
6,703
Our team created the technology behind CSF to give your local machines even more capacity for long workflows, large swarms and other tasks.
It's lossless, too. Meaning you don't lose any of the data you are caching by using it.
Enjoy your new free KV cache tokens!
Get up to 30% more KV cache using our lossless CSF version of @deepseek_ai V4.1-Flash-1017.
No drop in quality, just more room for you to do what you want to do!
Find the weights on our @huggingface page:
huggingface.co/local-inferen…
2
3
29
7,213
Local Inference Lab 🧪 retweeted
You can run it on @YourLocalAILab KK image if vLLM
Support already built in. Go to docker.local-inference-lab.a…
2
2
5
317
Local Inference Lab 🧪 retweeted
If you haven't heard of Local Inference Lab @YourLocalAILab you are not likely running local models as fast as you could be.
2
4
10
3,002
Local Inference Lab 🧪 retweeted
If you have Blackwell sm12x hardware, like a DGX Spark or Nvidia RTX 6000 GPU, you really need to be following the work of @YourLocalAILab.
QAD, an approach to better quality quantization developed by LIL, is now being recognized by @NVIDIAAI.
Check out their account here on X, website, Discord and engage in the growing community of people who are working together to get the most out of this amazing hardware.
We're on a mission to make it better for everyone and we're just getting started. 🦾
Want to run Qwen3.8 Flash at FP4 w/QAD to maintain high accuracy at low precision?
We have it optimized for RTX 6000, DGX Spark and other Blackwell sm12x hardware.
Check out our Qwen3.8-Flash-Next QAD weights on @NVIDIAAI's page at @huggingface 👇😎
huggingface.co/nvidia/Qwen3.…
2
6
19
7,460
Local Inference Lab 🧪 retweeted
GLM5.3-flash is cooking. Thanks and shout out to docker.local-inference-lab.a…! @YourLocalAILab
1
2
76
Local Inference Lab 🧪 retweeted
@YourLocalAILab Making moves with lossless compression now getting referenced by @NVIDIAAI . Proud of our team. Great job.👏
2
1
18
492