Pinned Tweet
I have spent the last year in the weeds with teams running open LLM inference on Modal.
Modal was already the best place to do this: truly elastic GPUs, fast cold starts, low-overhead routing, and deep observability.
Very excited to raise the bar even higher today.
Auto Endpoints package the Modal inference stack into one command:
modal endpoint create --name agent --model zai-org/GLM-5.2-FP8
6
10
93
13,297
the future of inference feels increasingly bespoke: continuously optimized for each model*hardware*workload. agents are what make this level of specialization economically viable
really impressive work from @DecagonAI! and cool to see @modal powering the experimentation here
every inference endpoint presents a different optimization problem, shaped by its model architecture, traffic patterns, hardware, and latency constraints.
we built an autoresearch loop to tackle these problems and carry lessons from one investigation into the next, with the broader goal of building systems that get better at research as they accumulate evidence.
2
4
100
5,972
are you getting it???
Today we're announcing that Modal has crossed $1B in annualized run rate, as well as raised a $750M Series D at a $15.75B valuation, led by @Accel.
Since our Series C, usage has tripled and we’ve grown more than 15x year-over-year.
We are deeply grateful to the developers and companies building on Modal. There's still so much more to build, and we can't think of a better time to be building software.
Made with AI
4
1
145
6,088
Very proud to have worked on this ambitious and total overhaul of how we route requests for Modal Functions. Check out this great read from @dshaar_ on how we shaved off ~80ms from our p50 function call latencies!
1
1
30
2,718
Many of us at Modal have pointed our harnesses to Kimi K3 (running on Modal!) and are _really_ happy with the results. Especially with the blazing fast performance thanks to our custom speculator.
Give it a try!
You can now run Kimi K3 in Codex on Modal. K3 leads open models on agentic coding benchmarks, with native vision and a 1M-token context window. Try it on our Shared Endpoint.
5
4
86
8,451
it's true, 460 tok/s!
Kimi K3 is live on Modal.
Moonshot has shipped the world's first open 3T-class model, and we're a day zero launch partner.
We trained a custom DFlash speculator for K3's novel architecture so you can run it faster, losslessly. The most capable open model we've worked with by far.
3
1
36
2,744
Deven Navani retweeted
Modal trained a DFlash speculator that's much faster than MTP, making it a great boost for inference speeds!
Inkling by @thinkymachines is now available on Modal, backed by a custom DFlash speculator for 67% higher throughput and interactivity.
Running on Modal Auto Endpoints with SGLang today.
14
44
445
60,318
Inkling is seriously impressive, try it out in Auto Endpoints! (with incredible performance enabled by our custom DFlash speculator)
A very big day for American open weights 🇺🇸 and we @modal are excited to support @thinkymachines with the launch
Inkling by @thinkymachines is now available on Modal, backed by a custom DFlash speculator for 67% higher throughput and interactivity.
Running on Modal Auto Endpoints with SGLang today.
2
46
3,239
We raised a Series C!
There's a lot left to build across Modal. We’re hiring, including on the inference team -- reach out if you’re interested!
Today we're announcing our Series C funding: $355M at a $4.65B valuation, led by some great investors @generalcatalyst and @Redpoint.
We've had insane growth in the last year, but we're still very early. So proud of the team and what we have built so far!
5
1
62
8,762
One of the many reasons I joined Modal is its culture — elite performance, technical taste, and an obsession with delighting customers.
This milestone is a testament to that culture, but we’re still in the early innings. With AI, code is no longer the bottleneck to building great products -- ideas are. Developers need a cloud platform that allows them to iterate quickly, with production-grade support for the entire AI lifecycle, from inference to training to sandboxes.
Modal will be this platform. We're hiring engineers across the entire stack -- reach out if you're interested!
It's true – @modal has raised a $87M Series B at a $1.1B valuation to advance the future of AI infrastructure.
Thank you to @Lux_Capital, @Redpoint, @AmplifyPartners, and others.
Now more than ever, AI demands a complete reinvention of traditional compute infrastructure
1
17
2,802
I’d love to learn more about how folks have deployed semantic search in production. What embedding and re-ranking models are you using? Which vector DB, if any? Are you fine-tuning the embedding model? DMs are open! and happy to share some learnings as well
1
6
1,734
Awesome work from @avnavani on fast course search for UCSD. Check it out!
Excited to release TritonSearch: the UCSD Course Search website! tritonsearch.xyz offers 🔥blazing fast🔥 text search speeds (<100ms) , making it easier than ever to find the courses you need @UCSanDiego.
9
1,913
Deven Navani retweeted
Long-term memory for AI chatbots
How do you make an AI chatbot smarter and more engaging? By giving it long-term memory of the entities it discusses with the user!
That's what @devennavani & I built for the @scale_AI hackathon (live today in @langchain). Honored to be one of the 8 finalists!
How it works:
1
6
66
32,377
Deven Navani retweeted
How do you make an AI chatbot smarter and more engaging? By giving it long-term memory of the entities it discusses with the user!
That's what @devennavani & I built for the @scale_AI hackathon (live today in @langchain). Honored to be one of the 8 finalists!
How it works:
33
60
769
197,892
Highly recommend giving Sieve a spin! @mokshith and @thpicy have built out a super compelling product that abstracts away all the gnarly bits of ML infra and makes several AI+video workflows possible with ~very~ little effort
1/ Today we’re excited to unveil Sieve (@sievedata), a cloud native platform for processing, searching, and running all sorts of AI models on video.
sievedata.com/blog/launch
6
Tweaking models to make incremental improvements to the SOTA is (relatively) low-hanging fruit, so it's hard to blame researchers (esp. grad students) who engage in this sort of research to collect citations and build their reputations.
arXiv has also replaced conferences as....
A lot of machine learning research has detached itself from solving real problems, and created their own "benchmark-islands".
How does this happen? And why are researchers not escaping this pattern?
A thread 🧵
ALT Image with the following text: Establish a research topic. For example: Detect COVID-19 from x-ray images with machine learning. First papers on this topic appear, and provide motivation for this research topic. A precedent has been established: The initial papers pave the way for further research papers on the same topic. The original motivation and assumptions for developing these predictive models are no longer questioned. More researchers publish data and predictive models. A community forms that cites and reviews each other. Certain datasets become benchmarks and certain predictive models become state-of-the-art. Predictive performance becomes the sole measure of progress, although improvements are becoming smaller. Actual progress in solving the initial research questions has become irrelevant, even discouraged. Decoupling from reality is complete.
2
1
the primary way to disseminate research ideas quickly. Arguably, ML conferences now just serve as brand stamps for researcher CVs, and with their deadlines every few months, they strongly incentivize surface-level research such as the benchmark chasing @ChristophMolnar described.
1
1
Yoshua Bengio has a great blog post from 2020 related to this and what a restructuring of incentives could look like:
yoshuabengio.org/2020/02/26/…
2