In Apache Spark 4.2, retrieval (NEAREST BY), governed semantics (metric views), and agent access (Spark Connect) run inside the engine, next to the data and under the same permissions, instead of in separate services. Download Spark 4.2 👉 spark.apache.org/downloads.h… #ApacheSpark #AI #GenAI

Felix Cheung (Apache Spark PMC, @nvidia) and @lisancao on running Spark SQL and DataFrames on GPUs with NVIDIA’s cuDF plugin, no job rewrites. Snapchat’s A/B experimentation pipeline: 10 PB of new data a day, processed with Spark. On GPUs, 70-90% lower cost. Also Spark Connect, including how a Spark Connect Gateway can sit in front of multiple servers, GPU ones included. 🎥 Full talk: piped.video/nL6NVivpIfQ #ApacheSpark #SparkConnect #cuDF

Spark SQL picks up several additions in Apache Spark 4.2, out now: SET PATH for object resolution, scripting cursors, QUALIFY for window results, SYSTEM.BUILTIN qualification, plus Theta sketches, time_bucket, and top-K max_by/min_by. More analytical work stays in SQL. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #SparkSQL #SQL

Apache Spark 4.2 is out. Python Data Sources let you build batch or streaming readers and writers in Python, no JVM or Scala, used through the standard spark.read interface. Spark 4.2 adds time and memory profiling so they're easier to tune and operate. 🔗 Download: spark.apache.org/downloads.h… #ApacheSpark #PySpark #Python

Auto CDC arrives in Spark Declarative Pipelines with Apache Spark 4.2: first-class SCD Type 1. Configure how change events update a target table, and Spark handles deletions and out-of-order events, in place of hand-written merge logic. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #CDC #DataPipelines #ETL

.@ApacheAirflow is a popular orchestrator, but it often has little visibility when a Spark job fails. At Airflow Summit 2026, @lisancao sat down with Andreas Neumann to discuss Spark Declarative Pipelines. Spark owns the pipeline graph, derives dependencies, and handles session lifecycle and writes, while Airflow orchestrates Spark as an application. 🎥 Watch the full conversation: piped.video/watch?v=dS5AwlC-… #ApacheSpark #ApacheAirflow #DataEngineering

Headed to #CommunityOverCode Glasgow? 🏴󠁧󠁢󠁳󠁣󠁴󠁿 Oct 11–14 features Apache Spark sessions on: 🔸 Resource auto-sizing 🔸 Spark 4.2 heterogeneous executors 🔸 CDC patterns 🔸 PySpark transpilation 🔸 Agent-native observability 🔗 See the schedule: communityovercode.apache.org… #ApacheSpark

Grounding agents in trusted data is a theme in Apache Spark 4.2: NEAREST BY for retrieval, metric views for governed numbers, Spark Connect for access. Retrieval and metrics live next to the data, under the same permissions. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #AIAgents #RAG #GenAI

Schema-on-Read Without Regrets: Lessons Learned from Apache Spark Variant Shredding in Production nitter.siddz.com/i/broadcasts/1nxnRBvAg…

Arrow-optimized Python UDFs are on by default in Apache Spark 4.2: existing UDFs take the faster columnar path with no rewrite, dropping the row-by-row JVM/Python serialization cost. Plus Pandas 3 and zero-copy handoff to Polars and @duckdb. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #PySpark #ApacheArrow #Python

Join us next Wednesday for Schema-on-Read Without Regrets! Geir Alstad (Gabler AS) and @newfront (@databricks) will walk through Spark Variant shredding in a real pipeline. 🗓️ Wed, Sept 30 · 8:00 AM PT 🎟️ Register: luma.com/open-z92a #ApacheSpark

Open Lakehouse + AI hits Bellevue in two weeks! 🎉 Felix Cheung (@nvidia) will cover GPU-accelerated Spark, Delta Lake, Iceberg, and Spark Connect. Micah Kornfield (@databricks) on a unified future for @DeltaLakeOSS and @ApacheIceberg, plus a panel on connecting the open lakehouse and AI ecosystem. 🗓️ Wed, Oct 7 🕜 5–9 PM PDT 📍 Databricks Bellevue Register: luma.com/open-qljr #ApacheSpark #ApacheIceberg

Apache Spark 4.2 is out, and it adds vector primitives to Spark SQL: distance and similarity functions, normalization, aggregation, and NEAREST BY (a top-K ranking join). Retrieval and recommendations run in SQL at Spark scale, next to your data. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #VectorSearch #SparkSQL

📣 Schema-on-Read Without Regrets: Lessons from Apache Spark™ Variant shredding in Production Join Geir E. Alstad (Gabler AS) + @newfront (@databricks) on Tuesday, Sept 30 at 8 AM PT. 🗓️ Learn how Variant shredding turns schema-less XML into fast streaming tables and a clean Kimball star schema, with real lessons on schema drift, reloads, and the Medallion Spoke pattern. 🎟️ Register: luma.com/open-z92a #OpenSource #ApacheSpark #Schema

Real-Time Mode now reaches PySpark in Apache Spark 4.2, out now: stateless streaming queries (no Python UDFs) at millisecond end-to-end latency, a path previously reachable mainly from the JVM. Stateful + UDF support is in progress for an upcoming 4.x (SPARK-54699). 🔗spark.apache.org/downloads.h… #ApacheSpark #StructuredStreaming #RealTime #PySpark #Streaming

In this video, @lisancao walks through Apache Spark 4.2, starting with metric views: define the measure once in YAML, store it in the catalog, and everything downstream hits the same number. 🔸 Real-time mode — Structured Streaming at millisecond latency; same DataFrame API. Now in PySpark for stateless queries 🔸 Vector search in Spark SQL — top-K nearest join on the arrays you already have. No extra vector store 🔸 Arrow-batch Python UDFs on by default, no code change 🎥 Watch: piped.video/4yEQmcpYrKc #ApacheSpark #Spark #PySpark #OpenSource