GPU tools @ Apple. Distributed systems before that. Side project: UchenML, an ML framework for hardware that has no business running models.
Silicon Valley
- Tweets 10,873
- Following 636
- Followers 2,881
- Likes 3,746
Pinned Tweet
UchenML links:
Browser demo - AlphaZero-style Dots: uchenml.tech/demos/dots
Overview: uchenml.tech/introducing-uch…
Using SIMD: uchenml.tech/simd-softmax-go…
Making inference fast: uchenml.tech/uchenml-optimiz…
1
10
3,024
In the lab, setting up new devices for our team, listening to talk by Alexandrescu. Fun.
piped.video/X_w_pcPs2Fk?si=cktS…
4
291
I am sitting in the morning
At the diner on the corner
I am waiting at the counter
For my corned beef hash and eggs.
I am sipping tea with lemon.
1
427
All the ML stuff I wrote on M4Max wrecks x86 machines. What’s the use of wider simd if you memory bandwidth is so low?
The memory unit economics of Apple Silicon are insane.
When you realise LLM inference is massively memory-bound (even at high concurrency!), you find that $ per memory bandwidth ($/GB/s) is the most important metric for inference.
M5 Ultra: $5499, 96GB @ 1.2TB/s, $4.5/GB/s
DGX Spark: $6950, 128GB @ 273GB/s, $25.5/GB/s
DGX Station: ~$150K, 252GB @ 7.1TB/s (HBM3e), $21.1/GB/s
M5 Ultra has 5.6x better $/GB/s than DGX Spark and 4.7x better $/GB/s than DGX Station.
I am surprised Apple didn't increase their prices more (maybe they will so might be a good time to buy...)
I expect 512GB M5 Ultra Mac Studio to start at $15899 (following the usual +$25/GB incremental memory price). That's a crazy good price for that much high speed memory.
With RDMA over Thunderbolt 5 you get single digit microsecond latency and you can connect up to 5 Thunderbolt 5 cables between Macs for 400Gb/s network bandwidth (same as ConnectX-7). Thunderbolt 5 cables are ~$25 each.
The main problem is the software on Apple Silicon is not great. There are about 30 competing inference engines, each supporting a small set of models, and quite unstable. The ecosystem around MLX is immature and limited (e.g. no Dynamo, LMCache, llm-d, ...). In our experience, more serious customers like Governments and Enterprises are put off by that and want something stable with an ecosystem of tools and software to run inference in production.
1
21
1,205
Drinking from the source - training data is coming from devices, listening in the middle. Uchen is under 200k in WASM - curios what the final binary size on a controller will be.
5
253
Rare footage of stochastic gradient descent.
Inside a human brain when reading a book.
Community note
This video shows time-lapse of cultured neurons connecting via growth cones in a lab dish, not processes in a living human brain while reading. Similar real clips by Dr. Lila Landowski show developmental wiring; this one's exact source is unconfirmed.
piped.video/watch?v=Rvmvt7…
lifo.gr/now/tech-scien…
1
11
518
Eugene Ostroukhov retweeted
I recently moved all of X web to Bazel and leveraging remote execution and remote caching on more than 1000 machines. It’s just so good! Unfortunately, not all models are good at it. (yet)
1
3
14
618
Sometimes I just give in and let my brain do what it wants instead of what I need. I really do not need a compile-time sprite library. I need to connect the controller to the Rokenbok truck I’m trying to make sentient and focus on training the model. But I want animations on that tiny screen. So that’s the weekend, apparently.
1
17
727
Eugene Ostroukhov retweeted
Infinite tokens and you still produce an electron app...
364
320
9,206
252,631
Part of my process is to have agents double- and triple- checking agents - asking them questions like “why is this becoming an issue only now” or “why are the numbers too good”.
1
249
I do not ask agents to deploy and such - instead, they maintain scripts I use. Every project has a “runbook” Python script that can push changes, wipe the db, etc. saves so much time and makes it reproducible.
Codex didn't publish your .env to cloudflare, YOU did. You are responsible for whatever clanker generates for you.
This is why YOU READ THE CODE. If you can't read the code, you hire someone who can.
Infra providers should stop bankrolling oopsies like this so people learn that when there is accountability, humans are needed.
1
9
625
Eugene Ostroukhov retweeted
I am so much more exhausted when I program with AI than without it. I reflected a bit, and I think I know why.
First, I think about how to solve a problem. This step is the same with and without AI. Then, with AI I have to write a prompt with a decent bit of precision and somewhat more long-term than if I just jumped into straight implementing it and figuring out the details as I go.
Then you start up the agent, and you immediately have some other work to do and you jump to another prompt for another task, where you also think really hard and try to formulate what you want quite well.
By the time you're done, first agent has some code for you to review, and you can't slack - you need to actually review it and be diligent about it. And when you're done - second task is there for your review. So really there's not time to rest.
When I program manually, I can decompress while typing the code in, I can rely on my muscle memory for typing and navigating the codebase, so there's periods of deep think and some relaxation. With AI, there's only deep think. And as soon as you try to think less - you open the doors for slop.
It's so tiring.
100
48
946
28,676
Growing up in a poor country, these stories are wild.
My son got an iPhone 6 with about a month of AppleCare left (so it was almost 3 years old). He plays Pokémon Go, and the battery was starting to give out. We took it to the Apple Store - and they just handed him a brand new iPhone 6. Kid was thrilled.
2
80
16,891
