Rendered at 19:47:07 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
w-m 1 days ago [-]
This is well-written. I could follow along quite nicely, from the setup through the bottlenecks and onto the resolution of the performance bug. Even the PRs are very pleasant to read: the majority of them is just a handful of changed lines with an added tests and a bit of documentation.
I was taken aback for a moment that this work originated from a report on StackOverflow. I had thought SO was effectively dead and abandoned by its community. But maybe I shouldn't project my own experience onto everyone else.
I’m not sure why it took me, a NumPy developer, looking at the benchmark numbers and saying “hmm, this is a bug”. But that is what it took. People are sometimes slow to treat behavior in dependencies like NumPy as bugs.
frollogaston 1 days ago [-]
I thought NumPy was already releasing the GIL. On regular non-free-threaded Python, you can run threaded parallel Numpy operations and have multiple cores doing 100%, I've relied on that. Maybe not the case with the operations this article focuses on (sin/cos).
quietbritishjim 10 hours ago [-]
Yes, numpy does release the GIL. But the code in question has multiple numpy calls, called in a loop:
sum((np.sin(np.cos(np.sin(np.cos(x + i)))).sum()
for i in range(n_loop)))
This is not like:
release GIL
# i = 0
compute y = x + 0 (numpy broadcasting sum)
compute z = np.cos(y) (elementwise)
...
add to running total
# i = 1
compute y = x + 1
...
reacquire GIL
Instead it is:
# i = 0
look up "+" operation
release GIL
compute y = x + 0
reacquire GIL
look up "np.cos" operation
release GIL
compute z = np.cos(y)
reacquire GIL
...
# i = 1
look up "+" operation
release GIL
compute y = x + 1
reacquire GIL
...
So there was work being protected by the GIL, that suddenly is exposed to lock contention with free threading.
Of course, without free threading, the lock contention would be way worse, but this time the GIL is the lock being contended. Numpy has to reacquire the GIL whenever it returns from a function call, and this expression is made up of multiple calls. To multithread effectively with numpy (in non-freethreading) you'd normally aim to vectorise into a small number of calls in big arrays.
The composition of +, then np.cos, etc. is not too bad if these are big arrays, but the problem is the pure Python iteration over the range which is, presumably, quite large. You could vectorise over the range:
There’s also the fact that INCREF and DECREF on shared objects is a lot more expensive than plain integer addition, so stuff becomes a bottleneck that was never a bottleneck. Kumar also fixed a bottleneck caused by a lock added only for safety on the free-threaded build. It’s hard to tell in advance than a fancy lock-free data structure is needed for something.
frollogaston 7 hours ago [-]
Ah ok, the times I did this and saw full CPU utilization was on pretty large arrays, so there was a lot less time spent in the GIL.
12 hours ago [-]
tialaramex 1 days ago [-]
> only acquire the lock when the flag needs to be updated
Unclear why you still need the lock here in that case. The idea that this flag may get updated during runtime and impacts how the software works when set seems to clash with the idea we need take no action having performed a relaxed (ie non-synchronising) load and seen it wasn't set at some previous time.
Maybe there's something I don't understand about these internals, which may be as simple as "It's just advisory so if we don't trace when we should no big deal".
pjmlp 1 days ago [-]
Nice to see the performance improvements work.
wiz21c 1 days ago [-]
I know this is more or less expected, but the improvement induced by adding a worker diminishes very rapidly... I guess it's not the cpython/numpy's fault but rather the CPU.
Even a tiny bit of serial instruction will limit the speed up
w-m 1 days ago [-]
The plot doesn't appear to be in Amdahl territory yet. The single-threaded time in the plot looks to be around 39 seconds. A perfect division into 32 workers without overhead would make it 39 / 32 = 1.22 seconds. With the multi-threaded workload being reported as 1.5 seconds in the text, there's still only .3 seconds of overhead + serial instructions that can't be parallelized.
Every doubling of the number of workers halves the execution time cleanly in the plot, from 40 seconds to 20 seconds to 10 seconds. Eyeballing this for 32 over 16 workers is difficult, but it still seems close to halving the total time once again. So there's not a lot of Amdahl flattening, it's just the plain physics of looking at a inverse-proportional curve.
artefactop 20 hours ago [-]
Great work!
dha111 1 days ago [-]
[flagged]
zbentley 1 days ago [-]
Maybe, but in this domain (compute-bound concurrency) there’s really no free lunch. The best languages in the space performance-wise are either full of sharp edges (C++, Go), or hard to use (Rust, Haskell). The friendliest languages require acrobatics and runtime improvements like the ones in TFA in order to perform better than abysmally.
FuckButtons 1 days ago [-]
Python is mostly glue around C/C++ numerical code for the things I’ve used it for, it’s the inverse of lua. In both cases you have a language that’s easy to manipulate tacked on to one that’s fast because time spent writing code is more expensive than time spent executing it.
pm90 1 days ago [-]
All of software is a hack in some way or another. You think of tradeoffs and make a decision. Theoretical purity is mostly an academic thing and Im sure you would love TCS.
bvan 1 days ago [-]
Perhaps, however, this hacky language is more productive in the real world than most. What’s the point, otherwise?
120394857 1 days ago [-]
The points:
- As the GP stated, the thrill of hidden bugs.
- Feeling productive due to fixing eternal issues.
- Getting paid to fix eternal issues.
- Feeling smart by talking about unnecessary issues.
- Writing a constant stream of PEPs to fix issues.
- Give conference talks about how you fixed issues.
- Give conference talks about how you will speed up Python by 5%.
frollogaston 11 hours ago [-]
You're saying these are the reasons people use Python?
I was taken aback for a moment that this work originated from a report on StackOverflow. I had thought SO was effectively dead and abandoned by its community. But maybe I shouldn't project my own experience onto everyone else.
Of course, without free threading, the lock contention would be way worse, but this time the GIL is the lock being contended. Numpy has to reacquire the GIL whenever it returns from a function call, and this expression is made up of multiple calls. To multithread effectively with numpy (in non-freethreading) you'd normally aim to vectorise into a small number of calls in big arrays.
The composition of +, then np.cos, etc. is not too bad if these are big arrays, but the problem is the pure Python iteration over the range which is, presumably, quite large. You could vectorise over the range:
but this is the start of a new conversation.Unclear why you still need the lock here in that case. The idea that this flag may get updated during runtime and impacts how the software works when set seems to clash with the idea we need take no action having performed a relaxed (ie non-synchronising) load and seen it wasn't set at some previous time.
Maybe there's something I don't understand about these internals, which may be as simple as "It's just advisory so if we don't trace when we should no big deal".
https://en.wikipedia.org/wiki/Amdahl%27s_law
Even a tiny bit of serial instruction will limit the speed up
Every doubling of the number of workers halves the execution time cleanly in the plot, from 40 seconds to 20 seconds to 10 seconds. Eyeballing this for 32 over 16 workers is difficult, but it still seems close to halving the total time once again. So there's not a lot of Amdahl flattening, it's just the plain physics of looking at a inverse-proportional curve.
- As the GP stated, the thrill of hidden bugs.
- Feeling productive due to fixing eternal issues.
- Getting paid to fix eternal issues.
- Feeling smart by talking about unnecessary issues.
- Writing a constant stream of PEPs to fix issues.
- Give conference talks about how you fixed issues.
- Give conference talks about how you will speed up Python by 5%.