Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You can release the GIL in extensions, and a lot of datascience libraries do that, so not all python computations are single threaded.


Support is partial and uneven. Moreover, you don't generally use a single datascience library, it's often a mix of pandas, numpy, scikit-learn, custom algorithms, custom optimization suite, and arrow. Letting individual library release the GIL is nice, but you need deep knowledge of those library to know what computation are thread-able or not.


In practice, for single-machine workloads, it's currently mostly numpy and/or whatever deep learning framework you use that does the number crunching.

This means that provided the code is operating on sufficiently large amounts of data (such that calls to numpy are each of sufficient duration), the multithreading in BLAS / Lapack within numpy usually give you weak scaling wrt to thread count without any tricks.

The issue however is that this require by hand making everything into structs of arrays from arrays of structs, removing as many iterations from python as possible, potentially balancing thread usage within python and within numpy etc. By this point IMO your "python" code looks more like Fortran or SQL with better string IO...


The number crunching part is already fast enough, however the aggregations, parsing and filtering that come beforehand is really really slow. This part is often done in pure python because it's often custom code tailored to the data you're manipulating.

We're not interested in scaling up the parts that are already fast, but the rather mundane, uninteresting work that come before.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: