Lightweight cloud compute platform for Python people.
Everywhere
- Tweets 1,727
- Following 54
- Followers 3,265
- Likes 2,090
Coiled retweeted
@natt941 made a good point on the podcast. "This message of 'Things are supposed to be delightful' is important to us."
And Matthew Rocklin agreed that the cloud "can be a delightful experience... Go play."
thenewstack.io/a-cloud-built…
2
5
1,072
Easily configure shared memory size for CLI jobs with `--docker-shm-size`.
Training PyTorch models on a GPU and need more memory? Ever run into "Error: No space left on device"?
Customize Docker shared memory size with `--docker-shm-size`.
docs.coiled.io/user_guide/cl…
ALT Carbon code snippet of the following CLI command: coiled run --docker-shm-size=3gb python train.py
2
1,433
🔨 Job setup option for Coiled Batch
Use `--host-setup-script` to configure your VM before your batch job starts.
Easily:
✅ Install dependencies
✅ Mount cloud storage
✅ Handle authentication
or any other setup your jobs need.
docs.coiled.io/user_guide/ba…
1,224
Coiled retweeted
~5 years ago I worked at a startup where we had multiple engineers screwing around for months with Terraform, Kubernetes, EKS, etc. just to get the same capabilities I got after an hour of playing around w/ Coiled.
Pretty cool.
2
1
3
545
Coiled retweeted
Coiled 2024 in Review
docs.coiled.io/blog/2024-eoy…
It’s the time when companies issue year-end summaries, acclaiming success (or not), and forecasting incredible growth for the next year (or not).
I thought I’d do something similar for Coiled. It’s been quite a year for us ...
5
27
2,020
Calculating quantiles, a common application in
#geospatial workloads, used to be slow due to GIL contention in NumPy.
The new implementation in @dask_dev + @xarray_dev is up to a hundred times faster and scales independently of the number of threads 🥳.
docs.coiled.io/blog/array-qu…
5
9
1,422
We're big fans of rich for a nice terminal experience, but have found sometimes folks log things even rich can't handle.
In the latest coiled=1.67.0 release, coiled logs automatically falls back to non-rich printing in these situations.
Release notes: docs.coiled.io/user_guide/ch…
ALT Rich terminal output from `coiled logs` CLI command
1
3
625
Coiled retweeted
New Post: Cloud Computing is Broken
matthewrocklin.com/cloud-is-…
Investor asks: "What's next for Data/Cloud Infrastructure?"
My answer: "Boring stuff. People struggle with basics."
Cloud feels like MP3 players before iPod. In theory everything is good. In practice adoption is low
1
3
22
1,933
We're now on Bluesky!
Should be pretty easy to find us, since bluesky lets us use our coiled.io domain as our handle ☀️
444
Coiled retweeted
Read about the latest improvement to GroupBy.map with Dask: xarray.dev/blog/dask-detrend…
Thanks to Patrick Hoefler of @CoiledHQ for the great work here!
1
6
31
3,678
Coiled retweeted
New Post: SLURM-Style Job Arrays on the Cloud
docs.coiled.io/blog/slurm-jo…
HPC Job scripts were the first form of parallelism I ever used as a graduate student. They're dead simple and accessible to almost anyone.
We replicated the API with Coiled. It feels pretty slick to me 🙂
1
5
16
912
Coiled retweeted
@CoiledHQ is amazing. If you want to have distributed compute and provisioned infrastructure from the code - its easy as that. Forget @ApacheSpark and @awscloud Sagemaker, EMR
1
4
228
Coiled retweeted
New blog post: Scale AI-based Data Processing EASY
The FineWeb-Edu dataset comes from processing 45TB (🤯) of FineWeb
And it uses a Language Model to classify the educational level of the text 😭😭
Still, we reproduced it in a few lines of code !
The key ? HF + Dask 😎
1
25
97
15,996
Coiled retweeted
Implemented @CoiledHQ into our product to offload data syncing from BigQuery to Neo4j 🤯
Works like butter 🧈
Now I don’t have to worry about scaling VMs dynamically to handle variable loads.
1
3
430
Coiled retweeted
We're to build a 100-TB scale geospatial benchmark suite
docs.coiled.io/blog/geospati…
We've seen an uptick in geospatial users and in challenges of the Xarray/Dask stack to scale beyond ~500-GiB.
This post presents a call for benchmark workloads.
2
12
44
4,619
Coiled retweeted
Arraylake and @CoiledHQ work great together! You can use Coiled to manage your cloud computing infrastructure with @dask_dev, and store your data as @zarr_dev in Arraylake.
We just added new a documentation page about our integration with Coiled.
docs.earthmover.io/integrati…
4
16
1,746
Run a Python script on a cloud GPU with one line of code.
Training a @PyTorch model training takes ~10 minutes and cost ~$0.12 on the NVIDIA T4 GPU on AWS. Coiled handles provisioning hardware, setting up drivers, and installing CUDA-compiled PyTorch.
docs.coiled.io/user_guide/gp…
2
702
Coiled retweeted
Dask DataFrame is now 20x faster. Some of most prominent changes include:
- Apache Arrow support in @pandas_dev
- Better shuffling algorithm for faster joins
- Automatic query optimization
Learn more: docs.coiled.io/blog/dask-dat…
ALT Bar chart showing a 20x improvement in Dask DataFrame performance with the addition of Arrow stings, more efficient shufffling, and a query optimizer.
8
30
2,830
Coiled retweeted
TPC-H Cloud Benchmarks: Spark, Dask, DuckDB, Polars
Across scales: 10 GiB, 100 GiB, 1 TiB, 10 TiB
Hardware: MBP and AWS
It was a fun experiment. No project wins uniformly. DuckDB and Dask do pretty well.
docs.coiled.io/blog/tpch.htm…
6
7
43
8,082
Coiled retweeted
Recommendation of the day: `coiled notebook start` to run a remote Jupyter Lab from big machines in cloud but with file sync that feel "local". Demo from @CoiledHQ piped.video/mibhDHYun0M
#python #jupyter
1
4
469