🇨🇦 Backend Brat. Distributed Diva. Relentless Learner.
Canada
- Tweets 115,917
- Following 370
- Followers 53,019
- Likes 10,530
Pinned Tweet
It’s important to fact check loud voices cuz there’s a lot of followers in this world. People who idolize you and your advice.
If someone’s promoting a way of working just to push a narrative that doesn’t actually work and people apply this advice today they can lose their jobs
8
19
250
12,525
Kelly Sommers retweeted
the compression time is actually significant portion of the request/response time. this implies that the coding harnesses should be compressing up to the previous message in the background/caching and then completing it when the new message comes in.
2
3
18
2,282
Kelly Sommers retweeted
I've uploaded a video of my "Vacuuming Large Tables" talk to piped.video/watch?v=ylqJ5CY2…. I did that talk a bunch of times but I think this was the only recording. If you're coming to my REPACK talk next week at PGConf EU, bonus points if you watch this one first. #postgres
1
4
1,182
Kelly Sommers retweeted
Our CEO, Max, officially announced that TiDB X will soon be open source.
Made with AI
3
6
29
2,308
Kelly Sommers retweeted
"In scale testing, one cluster ran over 6,700 pods across more than 100 node pools, with the busiest nodes holding 15 blocks (240 IPs) each."
planetscale.com/blog/planets…
4
2
65
5,400
Kelly Sommers retweeted
[new blog post]
Atlas Infinite: MongoDB's Disaggregated Architecture
muratbuffalo.blogspot.com/20…
16
74
6,810
Kelly Sommers retweeted
Today we're open-sourcing pg_vectorsearch, a new vector search extension for PostgreSQL.
The performance improvements over HNSW are pretty remarkable. At 100 million vectors:
◆︎ Faster queries, 4× less RAM: 6 ms vs. 8 ms at 95% recall, using 128 GB vs. 512 GB RAM. HNSW couldn't even build on 128 GB.
◆︎ 7× smaller indexes: 25 GB vs. 180 GB at 100M vectors.
◆︎ 50× faster index builds: 49 minutes vs. 42 hours on identical hardware.
And at 1 million vectors, PRISM delivers 15 ms queries at 99% recall on just 2 GB RAM, versus HNSW's 392 ms on 4 GB.
What leads to these improvements? Rather than traversing an HNSW graph with random memory accesses, our new PRISM index partitions vectors based on distance, compresses them using RaBitQ, and scans them efficiently using SIMD vectorization techniques. This takes better advantage of modern CPUs and memory locality.
This is actually our second open-source vector index. We learned a lot from building DiskANN-based pgvectorscale, including some important limitations of graph-based approaches. Those lessons convinced us there was a better architectural foundation, so we started over.
pg_vectorsearch is still in alpha, with more engineering ahead. But we believe fast, scalable vector search should live alongside your relational data, without requiring a separate vector database.
Released under the permissive PostgreSQL License, just like our pg_textsearch extension for BM25 keyword search. Together, they provide the building blocks for hybrid vector + keyword search in PostgreSQL.
Feedback and contributions welcome!
Code and blog post below.
⚡️⚡️⚡️
38
98
858
39,404
🔥 I’ve seen a lot of teams struggle with expensive and slow MSSQL offerings on Azure so this will be fire.
4
2
82
11,003
Kelly Sommers retweeted
Engineering introduction to distributed query execution and how it is implemented by DataFusion distributed.
datadoghq.com/blog/engineeri…
3
22
202
8,068
Kelly Sommers retweeted
AWS's own docs say gp3 volumes can drop below 90% of provisioned IOPS 1% of the time.
That's 86 hours a year, per volume. Spread a database across 768 of them and there's a 99.65% chance one is hurting production right now.
planetscale.com/blog/the-rea…
7
8
155
21,857
In case someone’s curious where I left off with my fork of the original Rails code
I fixed some sloppy cache code that hits the DB on every cache hit (yes really)
Pretty confident most of the issues are just app code and you could get it down even more.
11
9
249
28,005
This is how real experience speaks.
Not pushing a narrative even though there’s an opportunity to make yourself look great with a hot take.
A measured and responsible answer.
my truthful answer is that i would never presume what the problems are. i have not seen the inside of github's architecture since 2019. i know they have good folks working there. i know that type of scale coming so quickly is very hard to react to when you run at the scale GiHub runs at. nothing is really that elastic, and orders of magnitudes more traffic showing up very quickly is hard to get in front of. I've worked at extreme scale before and seen companies spend three months just preparing for events that will increase their traffic by 10% or 20% percent, which is nothing compared to what GitHub's facing now.
5
13
373
15,490
Thanks for tolerating my spam the last few days. I really enjoy the community along the way but I know for some who aren’t interested I’m tweaking your unfollow button 😂
10
1
75
3,156
I’ve been asked why I was so passionate to dig into this system.
I am myself experimenting with how do I accelerate and maybe read less code but still ship code confidently. This is a big shift for me and many.
But terrible AI prompts and no review _today_ will get you fired.
8
11
163
7,051
