> spark is dramatically overkill for the data sets dask is meant to work on
I disagree with this, I have used spark standalone/single node both locally and in the cloud for numerous use cases because you can easily cook up spark sql or pyspark cleaning and ETL scripts for small data (a couple TB) and reach high throughput (>260mb/s). I am also someone who reaches for dask (and prefect) first for small data but in cases where the data is medium sized (100gb-10tb) I always reach for spark first, its simply easier to write fast code with spark (especially spark sql) without needing to think hard about optimizations. I also have not run into any operational issues running a spark standalone instance for a few hours or days to process small data I would use dask for.
I reach for dask first for most small data (<100gb) because its better integrated with python libraries like prefect that improve the dev experience.
Spark seems like it will be good, but always disappoints.
I doubt know much about spark, so I may get this wrong, but I feel that they need to greatly improve both:
1) the automatic configuration of single node systems
2) the changing of config throughout the processing
The fact that I don't even really know if this is the problem however is the real problem.
I get that setting stuff up for a huge network of computers may be challenging, but standard defaults to use all processors on a single machine should be automatic.
Spark pisses me off that the defaults for a 1TB RAM, 20 core machine as a single node can't handle reading a 20GB parquet file with one line.
And further, checking substrings can throw OOM errors in the next step without adjusting the config of worker memory, etc.
Worst of all, a 40 MB (!) file after filtering and processing can take like 10 minutes to save because of all the parallelism config. It's off course instantaneous on pandas.
They really need to fix the experience for single machines (e.g. just like a small 18 core, 512 GB ram system), so that people can write packages that work for anyone, but with a small yaml flag, it can be sent to a cluster.
I disagree with this, I have used spark standalone/single node both locally and in the cloud for numerous use cases because you can easily cook up spark sql or pyspark cleaning and ETL scripts for small data (a couple TB) and reach high throughput (>260mb/s). I am also someone who reaches for dask (and prefect) first for small data but in cases where the data is medium sized (100gb-10tb) I always reach for spark first, its simply easier to write fast code with spark (especially spark sql) without needing to think hard about optimizations. I also have not run into any operational issues running a spark standalone instance for a few hours or days to process small data I would use dask for.
I reach for dask first for most small data (<100gb) because its better integrated with python libraries like prefect that improve the dev experience.