Francesc Alted @FrancescAlted@masto.social@FrancescAlted

Creator and BDFL of Blosc/@Blosc2. Co-founder and CEO of ironArray.io. Love Python & C. github.com/FrancescAlted

 Castelló de la Plana
ironarray.io/
Joined January 2012
138 Photos and videos
📢 Python-Blosc2 4.14 is out! A common interface to stream, query & slice remote Parquet, #PyTables, #HDF5, #Zarr & #Blosc2 data over HTTP/S3: ⚡️ Zero full downloads ⚡️ Cloud index pushdown ⚡️ DuckDB/Polars streaming! Blog 👇 blosc.org/posts/remote-local… Stream Smarter 🚀

Common Interface for Remote and Local Datasets

Python-Blosc2 4.14 brings a common streaming and caching interface to remote and local Parquet, Zarr, HDF5, PyTables, and Blosc2 datasets.

blosc.org
1
1
56
We are making a lot of progress on accessing remote PyTables tables via Blosc2 and fsspec. Stay tuned!
5
150
📢 Python-Blosc2 4.13.1 is out! Continuing our "not an island" mood: stream remote datasets (HDF5, Zarr, B2Z) over HTTP/S3 lazily, and compute across formats in one expression: expr = b2z + zarr * h5 blosc.org/python-blosc2/guid… Compress Better, Compute Bigger, Download Faster 🚀
1
2
225
Release notes at: github.com/Blosc/python-blos…
1
31
📢 Python-Blosc2 4.13.0 is out! One lazy API for remote data: browse B2Z/Zarr/HDF5 hierarchies over S3/GCS/HTTP, slice only what you need, cache in RAM or on disk, and export portable snapshots. b2view now explores remote containers👇 github.com/Blosc/python-blos…
5
132
📢 Caterva2 2026.9.2 is out! 🚀 Intelligent compute & peer caching for Blosc2 & HDF5: ✅ C2Cache peer caching ✅ Blosc2 CTable (.b2z) support ✅ Virtual roots (.b2z/.h5) ✅ Concurrent chunk writes ✅ Embedded JupyterLite github.com/ironArray/Caterva… Share Faster, Compute Bigger 🚀
1
86
📢 Python-Blosc2 4.12.0 is out! Remote arrays work over S3/GCS/HTTP via fsspec. lazy=True fetches only the blocks a slice needs and reuses a validated cache — 5–17x faster in S3 benchmarks. Enjoy data 🚀 github.com/Blosc/python-blos… #Python #DataScience #HPC #CloudComputing
77
💡 Blosc2 tip: repeated text? store it as dictionary(). One int32 code per row plus a single copy of each value, so grouping and filtering are integer work. Massive gains in frequently used string operations. blosc.org/python-blosc2/guid… Enjoy data! 🚀 #Python #DataScience
1
58
📣 Python-Blosc2 4.11.0 continues aligning with Arrow conventions. CTable nullability on a validity mask instead of a sentinel value ✨️ + Wheels are now abi3 compliant w/ CPython 3.11+, including 3.15 (tested) 🚅 github.com/Blosc/python-blos… Compress Better, Compute Bigger 🚀

Releases · Blosc/python-blosc2

A high-performance library for compressed ND arrays and columnar tables, with compute and indexing engines - Blosc/python-blosc2

github.com
4
146
💡 Blosc2 tip: @blosc2.jit can compile complex control flow with loops and conditionals. A Mandelbrot escape loop written plainly: ~155x faster than plain Python, ~2.3x than hand-vectorized NumPy. blosc.org/python-blosc2/guid… Compress Better, Compute Bigger 🚀 #Python #Blosc2
1
91
💡 Blosc2 tip: for text you can't bound, use utf8(). title: str = blosc2.field(blosc2.utf8()) string(max_length=200) costs 800 B/row on every read. utf8() rows cost what they weigh: ~4× less peak memory. blosc.org/python-blosc2/guid… Compress Better, Compute Bigger 🚀 #Blosc2
1
119
Most compression libraries ask you to move in. We think that's backwards. Blosc2 4.9.1: DuckDB, Polars, pandas 3 and PyArrow read a CTable directly via Arrow's PyCapsule protocol — no conversion step, no copy. Cooperation, not completeness. blosc.org/posts/not-an-islan… Enjoy data!

Not an island: bringing compression to the tabular ecosystem

A compression library can go one of two ways: try to be everything (its own dataframe, its own query engine, its own format that nothing else reads), or be a fast, compact layer that slots underneath

blosc.org
1
65
💡 Blosc2 tip #9: mmap read-only opens — skip the syscalls. >>> arr = blosc2.open(path, mmap_mode="r") Maps once, reads pages directly. ~15% faster, same memory. Free win, scales with concurrent readers 🚀 blosc.org/python-blosc2/guid… #Python #DataScience
1
67
💡 Blosc2 tip: `where=` for filtered reductions. t["temperature"].sum(where=t.region == 3) ~1.9× faster, ~7× less memory than mask-then-index. blosc.org/python-blosc2/guid… Compress Better, Compute Bigger 🚀 #Python #DataScience
1
57
💡 Python-Blosc2 tip #7: reducing a column? Don't slice it first. t["val"].sum() reduces chunk-wise over the compressed column instead of materialising it — 1.7× faster and ~12× less peak memory on a 50M-row column. blosc.org/python-blosc2/guid… Compress Better, Compute Bigger 🚀
1
72
💡 Python-Blosc2 tip #6: generate arrays with DSL kernels. A @blosc2.dsl_kernel function handed to blosc2.lazyudf() compiles to native code and fills an NDArray chunk by chunk, in parallel. At 200M elements, ~3.6x faster, ~2x less memory than NumPy. blosc.org/python-blosc2/guid…
3
158
💡 Python-Blosc2 tip #5: sorted top-k? Stream it from a FULL index CTable.sort_by(view=True)[:k] and NDArray.iter_sorted(start=-k) read just that slice from the index sidecar instead of sorting everything: up to ~74× less time, ~193× less peak memory. blosc.org/python-blosc2/guid…
1
96
💡 Python-Blosc2 tip #3: align your reads with the double partition A chunk-aligned read decompresses 1 chunk instead of 2, and chunk-aligned slice() copies chunks as-is with no decompression at all. Same principle at block level. blosc.org/python-blosc2/guid… Enjoy data! 🚀
1
88
💡 Python-Blosc2 tip #2: many readers, one file? Open with mmap_mode="r" in every reader — all share one set of mapped pages instead of a syscall + copy per read. 8 concurrent readers: ~4.5x faster, half the CPU, still ~one copy of the file in RAM. blosc.org/python-blosc2/guid…
3
89
💡 Python-Blosc2 tip #1: skip the NumPy detour. blosc2.linspace(0, 1, N) fills a compressed array chunk by chunk — at 200M float64, ~25x less peak memory than asarray(np.linspace(...)), comparable speed. Same for arange() and fromiter(). blosc.org/python-blosc2/guid… Enjoy data! 🚀
3
263
Load more