Skip to content

Implement PyDenseArray - #821

Open
JamesWrigley wants to merge 1 commit into
JuliaPy:mainfrom
JamesWrigley:pydensearray
Open

JamesWrigley wants to merge 1 commit into
JuliaPy:mainfrom
JamesWrigley:pydensearray

Conversation

@JamesWrigley

Copy link
Copy Markdown
Contributor

This is a minimal array type to allow dispatching to things like BLAS functions that require DenseArray, inspired by the idea proposed in #319 (comment). Example performance improvement:

julia> using PythonCall, BenchmarkTools, LinearAlgebra

julia> @py import numpy as np

julia> x_py = np.random.rand(1000, 1000);
                                                                                                                                                                                                                                               
julia> x1 = pyconvert(PyArray, x_py.T);
                
julia> x2 = pyconvert(PyDenseArray, x_py);

julia> @benchmark mul!(out, x1, x1) setup=(out=zeros(1000, 1000))
BenchmarkTools.Trial: 20 samples with 1 evaluation per sample.
 Range (min  max):  253.484 ms  264.718 ms  ┊ GC (min  max): 0.00%  0.00%
 Time  (median):     259.075 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   258.614 ms ±   2.880 ms  ┊ GC (mean ± σ):  0.00% ± 0.00%

  ▁    ▁ █     ▁      █   ▁ ▁  ▁ ▁ █ █   ▁▁   ▁  ▁            ▁  
  █▁▁▁▁█▁█▁▁▁▁▁█▁▁▁▁▁▁█▁▁▁█▁█▁▁█▁█▁█▁█▁▁▁██▁▁▁█▁▁█▁▁▁▁▁▁▁▁▁▁▁▁█ ▁
  253 ms           Histogram: frequency by time          265 ms <

 Memory estimate: 0 bytes, allocs estimate: 0.

julia> @benchmark mul!(out, x2, x2) setup=(out=zeros(1000, 1000))
BenchmarkTools.Trial: 429 samples with 1 evaluation per sample.
 Range (min  max):  9.034 ms   13.821 ms  ┊ GC (min  max): 0.00%  0.00%
 Time  (median):     9.213 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   9.270 ms ± 339.861 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%

    ▅▁  ▁▄▃▃▅▄▆█▂                                              
  ▄▄██▇▇█████████▃▅▃▃▃▂▄▁▄▆▆▅▄▄█▇▂▄▁▂▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▂ ▄
  9.03 ms         Histogram: frequency by time        9.97 ms <

 Memory estimate: 0 bytes, allocs estimate: 0.

I think this makes more sense than a DenseView type in a separate package because this way in v1 we can make it the default instead of PyArray when converting contiguous arrays. Which would be very nice for performance :) It ended up being a fairly small amount of code since it's such a restricted type, so hopefully it's not too much of a maintenance burden.

Written with help from Claude 🤖

This is a minimal array type to allow dispatching to things like BLAS functions
that require `DenseArray`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant