Python 3.13 RC1 - a Quick CPU Benchmark

Python 3.13 is due to be released in October, yet the first release candidate was published earlier in August. Last year, I did a quick CPU bound benchmark of version 3.12 using Mandelbrot set calculation.

With all files in place, I downloaded version 3.13RC from the official web site and tested 3 versions of Python on my M1 Mac Book Pro:

user@users-MacBook-Pro py_chat_ui % python --version
Python 3.11.9
user@users-MacBook-Pro py_chat_ui % python3 --version
Python 3.12.5
user@users-MacBook-Pro py_chat_ui % python3.13 --version
Python 3.13.0rc1

Here're the execution time (in seconds) for the implementation relying on NumPy (versions 3.11, 3.12 and 3.13RC1 correspondingly):

user@users-MacBook-Pro mandelbrot % python mandelbrot.py   
1 Execution Time: 6.305059909820557                   78513419
2 Execution Time: 6.282307863235474                   78513419
3 Execution Time: 6.473501920700073                   78513419
user@users-MacBook-Pro mandelbrot % python3 mandelbrot.py
1 Execution Time: 5.418003082275391                   78513419
2 Execution Time: 5.412122011184692                   78513419
3 Execution Time: 5.434246778488159                   78513419
user@users-MacBook-Pro mandelbrot % python3.13 mandelbrot.py
1 Execution Time: 7.197513818740845                   78513419
2 Execution Time: 7.212265968322754                   78513419
3 Execution Time: 7.200297832489014                   78513419

And the results for the pure Python implementation:

user@users-MacBook-Pro mandelbrot % python mandelbrot_pure.py
1 Execution Time: 41.18416976928711                   78513425
2 Execution Time: 41.16466403007507                   78513425
3 Execution Time: 41.148504972457886                   78513425
4 Execution Time: 41.55486297607422                   78513425
user@users-MacBook-Pro mandelbrot % python3 mandelbrot_pure.py
1 Execution Time: 49.806406021118164                   78513425
2 Execution Time: 49.485753774642944                   78513425
3 Execution Time: 49.52305006980896                   78513425
4 Execution Time: 49.57118225097656                   78513425
user@users-MacBook-Pro mandelbrot % python3.13 mandelbrot_pure.py
1 Execution Time: 41.07340693473816                   78513425
2 Execution Time: 41.08624267578125                   78513425
3 Execution Time: 41.09266400337219                   78513425
4 Execution Time: 41.1431610584259                   78513425

Frankly speaking, I am confused with the inconsistency in the results - newer doesn't mean better :)

P.S. >

For the reference, running Mandelbrot calculation with JiT compiled Dart program and AoT compiled C version:

user@users-MacBook-Pro mandelbrot % dart mandelbrot.dart
1  Execution Time: 0.476                       78513425
2  Execution Time: 0.484                       78513425
3  Execution Time: 0.475                       78513425
user@users-MacBook-Pro mandelbrot % gcc -o mandelbrot mandelbrot.c -Ofast
user@users-MacBook-Pro mandelbrot % ./mandelbrot 
1 Execution Time: 0.256706           79394433
2 Execution Time: 0.234396           79394433
3 Execution Time: 0.234862           79394433

P.P.S. > Here's the GitHub repo with the same benchmark implemented in different languages.

P.P.P.S. > Here's last year's post comparing Python, Numba and Mojo.

Comments

[Nir A]

Modifying mandelbrot_0() to the following python aware code, cuts down running time to 6.8 seconds per run. decorating both functions with @numba.jit cuts it down to 0.26 seconds:

def mandelbrot_0(c: complex) -> int:
... z = 0j
... for i in range(MAX_ITERS):
....... z = z * z + c
....... if (z * z.conjugate()).real > 4.0:
........... return i
... return i

Runtime with python aware code decorated with @numba.jit:

1 Execution Time: 0.3756251335144043 78513425
2 Execution Time: 0.2627451419830322 78513425
3 Execution Time: 0.26718902587890625 78513425
4 Execution Time: 0.2661020755767822 78513425

Runtime with python aware code:

1 Execution Time: 6.8413779735565186 78513419
2 Execution Time: 6.812233924865723 78513419
3 Execution Time: 6.828969955444336 78513419
4 Execution Time: 6.858006000518799 78513419

Runtime of original code:

1 Execution Time: 41.62785005569458 78513425
2 Execution Time: 41.481120109558105 78513425
3 Execution Time: 41.68783092498779 78513425
4 Execution Time: 41.20909810066223 78513425