Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression
- Tweets 360
- Following 26
- Followers 8,601
- Likes 20
Apache Parquet retweeted
Turns out @ApacheParquet Bloom filters are better than I think many people understand. Trevor Hilton found that for a cost of 2K-8K per row group on high cardinality predicate columns, you can filter all but the exact row group of interest.
Great writeup by @InfluxDB developer Trevor Hilton on using @ApacheParquet's Bloom Filters: influxdata.com/blog/using-pa…
10
63
9,871
Apache Parquet retweeted
To anyone who thinks @ApacheParquet is dead, it it showing renewed signs of life 🌹
It’s happened! The @ApacheParquet Java implementation repo I now called parquet-java. Thank you @andrewlamb1111 for the nudge! This further clarifies that Parquet is used far beyond the Hadoop ecosystem. Maybe whoever created this repo could have thought of this to start with.
4
1
37
5,281
Apache Parquet retweeted
I tweeted this ten years ago today. At the time I didn’t quite realize how much impact this little side project would have. To ten years of Parquet! Thanks to all the people who came along for the ride.
1
8
70
Apache Parquet retweeted
It’s happened! The @ApacheParquet Java implementation repo I now called parquet-java. Thank you @andrewlamb1111 for the nudge! This further clarifies that Parquet is used far beyond the Hadoop ecosystem. Maybe whoever created this repo could have thought of this to start with.
4
22
7,443
Apache Parquet retweeted
PSA: If you use the page-level statistics in @ApacheParquet please chime in on JIRA: issues.apache.org/jira/brows…
3
3
Apache Parquet retweeted
Last speaker on the #europython's scientific room before lunch is Peter Hoffmann talking about#Pandas and #Dask to work with large datasets in @ApacheParquet.
2
16
35
Apache Parquet retweeted
Have a look at the @ApacheFlink bucketing sink rework for the upcoming release and the Parquet writer ;)
3
2
4
Apache Parquet retweeted
@StackOverflow @ApacheSpark Can someone answere this -> why is @ApacheParquet format faster than other columnar storage like hbase, kudu etc? stackoverflow.com/q/48761227…
1
3
5
Apache Parquet retweeted
2nd #PyDataLDN #keynote - @holdenkarau & @BooProgrammer walk us through a zoo of #tools for #BigData & #distributed #data in #Python: #Apache #Spark, #PySpark, #Arrow, #Beam, #Parquet & #Dask
@ApacheSpark @ApacheArrow @ApacheBeam @ApacheParquet @dask_dev
#PyData @PyData @NumFOCUS
10
24
Apache Parquet retweeted
Join the #GPU accelerated #analytics and #ML revolution. @ApacheArrow @ApacheParquet and @gpuoai #GTC18
6
10
Apache Parquet retweeted
Great benchmark between @ApacheParquet on #hdfs and @ApacheKudu blog.clairvoyantsoft.com/gui…
In short kudu is faster than Parquet for random access Querys like CRUD operations but slower for analytics queries.
10
13
Apache Parquet retweeted
If you’re a company using open source projects and not sure how to contribute, a release engineer would be a tremendous help. It’s hard to do this properly part time. I have a specific project in mind, if you need a hint.
6
6
Apache Parquet retweeted
You do not need Spark to create @ApacheParquet files, you can use plain Java and it can even fit in AWS Lambda for a serverless solution:
engineering.opsgenie.com/ana…
7
14
Apache Parquet retweeted
Learn how to use hive views for advanced schema evolution #hive blog.nuvola-tech.com/2017/02…
1
3
Apache Parquet retweeted
I wonder if we have @ApacheParquet in #rstats
Working with a 10Gig csv data. Pandas read_csv took 16mins to load the csv into memory. Converted to @ApacheParquet with @ApacheArrow. It took 30 secs to read into pyarrow table and 16 sec to convert to pandas dataframe.
16mins => 46sec!
tech.blue-yonder.com/efficie…
1
6
7
Apache Parquet retweeted
Also the file size went down from 10Gigs to 3Gigs without any compression.
1
6
18
Apache Parquet retweeted
Working with a 10Gig csv data. Pandas read_csv took 16mins to load the csv into memory. Converted to @ApacheParquet with @ApacheArrow. It took 30 secs to read into pyarrow table and 16 sec to convert to pandas dataframe.
16mins => 46sec!
tech.blue-yonder.com/efficie…
13
142
502
Apache Parquet retweeted
Come hear me talk about @ApacheArrow and @ApacheParquet at #NABDConf in Palo Alto next Tuesday! nitter.siddz.com/jqcoffey/status/927859…
1
4
8
Apache Parquet retweeted
At @ucc_bdcat today in #Austin presenting our work with @pbr_wur on managing #agri #genomic #bigdata with @ApacheSpark and @ApacheParquet
1
8
9