When you want to process 450Gb/1billion rows of data we think in all the directions like PySpark, Bigquery and etc. If someone says it can be processed with one Python package(DuckDB) without using/installing any fancy tools can you believe it? That’s what João Pedro did and explained in this article.
My First Billion (of Rows) in DuckDB | by João Pedro | May, 2024 | Towards Data Science
First Impressions of DuckDB handling 450Gb in a real project
towardsdatascience.com
Top comments (0)