By: Tim Besard

Re-posted from: https://juliagpu.org/post/2023-03-03-metal_0.2/index.html

Metal.jl 0.2 marks a significant milestone in the development of the Metal.jl package. The release comes with initial support for the Metal Perform Shaders (MPS) framework for accelerating common operations like matrix multiplications, as well as various improvements for writing Metal kernels in Julia.

Metal Performance Shaders

Quoting the Apple documentation, The Metal Performance Shaders (MPS) framework contains a collection of highly optimized compute and graphics shaders for use in Metal applications. With Metal.jl 0.2, we have added initial support for this framework, and used it to accelerate the matrix multiplication operation:

julia> n = p = m = 2048
julia> flops = n*m*(2p-1)
17175674880julia> a = MtlArray(rand(Float32, n, p));
julia> b = MtlArray(rand(Float32, p, m));
julia> c = MtlArray(zeros(Float32, n, m));julia> bench = @benchmark Metal.@sync mul!(c, a, b)
BenchmarkTools.Trial: 518 samples with 1 evaluation.
 Range (min … max):  9.366 ms …  13.354 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     9.629 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   9.646 ms ± 192.169 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%               ▃▂▅▅▆▆▆▇█▇▇▆▅▄▄▁▁ ▁
  ▄▁▄▄▄▄▆▆▆▄▄▁▇█████████████████▄█▄▁▆▁▄▁▆▁▇▁▄▄▁▁▄▄▇▁▄▆▄▁▁▁▁▁▄ █
  9.37 ms      Histogram: log(frequency) by time      10.1 ms < Memory estimate: 352 bytes, allocs estimate: 12.julia> flops / (minimum(bench.times)/1e9)
1.83e12

The benchmark above shows that on an 8-core M1 Pro matrix multiplication now reaches 1.8 TFLOPS (out of the 2.6TFLOPS of theoretical performance). The accelerated matrix multiplication is available for a variety of input types, incuding mixed-mode operations, and as shown above is integrated with the LinearAlgebra.jl mul! interface.

Of course, the MPS framework offers more than just matrix multiplication, and we expect to support more of it in the future. If you have a specific operation you would like to use from Julia, please let us know by opening an issue on the Metal.jl repository.

GPU profiling support

To support the development of Metal kernels, Max Hawkins has added support for GPU profiling. Similar to how this works in CUDA.jl, you can run code under the Metal.@profile macro to record its execution. However, this does first require setting the METAL_CAPTURE_ENABLED environment flag before import Metal.jl:

julia> ENV["METAL_CAPTURE_ENABLED"] = 1julia> using Metaljulia> a = mtl(rand(1024, 1024))
julia> Metal.@profile sum(a)
[ Info: GPU frame capture saved to jl_metal.gputrace/

The resulting capture can be opened with Xcode, presenting a timeline that's similar to other profilers:

Other improvements

• Julia 1.9 is supported, but requires an up-to-date macOS version (issues have been encountered on macOS 12.4);

• An mtl function has been added for converting Julia arrays to Metal arrays, similar to the cu function in CUDA.jl;

• Multiple GPUs are supported, and the device! function can be used to select one;

• Coverage for SIMD Group functions has been improved, so it's is now possible to use simdgroup_load, simdgroup_store, simdgroup_multiply, and simdgroup_multiply_accumulate in kernels functions.

Future work

Although Metal.jl is now usable for a variety of applications, there is still work to be done before it can be considered production-ready. In particular:

• there are known performance issues with mapreduce, and other operations that realy on CartesianIndices;

• the libcmt wrapper library for interfacing with the Metal APIs is cumbersome to use and improve, and we are looking into native ObjectiveC FFI instead;

• the MPS wrappers are incomplete, and similar to the Metal APIs requires a replacement to libcmt to be improved;

• support for atomic operations is missing, which is required to implement a full-featured KernelAbstractions.jl back-end.

Once (most of) these issues are addressed, we should be able to release Metal.jl 1.0.

By: Tim Besard

Re-posted from: https://juliagpu.org/post/2023-02-08-oneapi_1.0/index.html

The release of oneAPI.jl 1.0 adds integration with the oneAPI Math Kernel Library (oneMKL) to accelerate linear algebra operations on Intel GPUs. It also brings support for Julia 1.9 and Intel Arc GPUs.

oneMKL integration

oneAPI.jl now uses the Intel oneAPI Math Kernel Library (oneMKL), automatically downloaded as part of oneAPI_Support_jll.jl, to accelerate a great number of BLAS and LAPACK operations on Intel GPUs. Similar to how it is implemented in our other GPU back-ends, these wrappers are available at different levels of abstraction.

At the lowest level, we use a C library that wraps the oneMKL C++ APIs. For example, the oneapi::mkl::blas::column_major::gemm function for matrix-matrix multiplication is wrapped by the C functions onemklSgemm, onemklDgemm, etc. These wrappers are used to implement low-level methods like oneMKL.gemm!:

julia> using oneAPIjulia> A = oneArray(rand(Float32, 2, 3));
2×3 oneMatrix{Float32, oneAPI.oneL0.DeviceBuffer}:
 0.44302   0.125576  0.859145
 0.674291  0.428346  0.0400119
julia> B = oneArray(rand(Float32, 3, 4))
3×4 oneMatrix{Float32, oneAPI.oneL0.DeviceBuffer}:
 0.592748   0.529413   0.0323396  0.659528
 0.22489    0.0872259  0.253291   0.376519
 0.0121506  0.591135   0.706755   0.751686
julia> C = similar(B, (2, 4));julia> oneMKL.gemm!('N', 'N', true, A, B, true, C)
2×4 oneMatrix{Float32, oneAPI.oneL0.DeviceBuffer}:
 0.301279  0.753365  0.65334   0.985274
 0.496501  0.417994  0.158581  0.63607julia> Array(C) ≈ Array(A) * Array(B)
true

Of course, these low-level functions aren't very user-friendly, so we also integrate with Julia's standard libraries where possible:

julia> A = oneArray(rand(Float32, 2, 3));
julia> B = oneArray(rand(Float32, 3, 4));julia> using LinearAlgebra
julia> C = A * B;julia> Array(C) ≈ Array(A) * Array(B)
true

The most frequently used oneMKL BLAS functions have been wrapped and integrated with Julia’s standard linear algebra libraries. If you run into a missing function, please file a request to add it, or take a look at the source and contribute to oneAPI.jl! The current state of the wrappers should make it easy to extend their functionality, as well as form a good basis for integrating with other libraries like oneDNN.

Intel Arc support

The new Arc series of discrete Intel GPUs are now fully supported by oneAPI.jl. These GPUs offer a significant performance improvement over their integrated predecessors:

julia> using oneAPI
julia> oneAPI.versioninfo()
1 device:
- Intel(R) Arc(TM) A770 Graphics [0x56a0]julia> T = Float32;
julia> n = p = m = 2048;
julia> a = oneArray(rand(T, n, p));
julia> b = oneArray(rand(T, p, m));
julia> c = oneArray(zeros(T, n, m));julia> using BenchmarkTools, LinearAlgebra
julia> bench = @benchmark oneAPI.@sync mul!(c, a, b)
BenchmarkTools.Trial: 1510 samples with 1 evaluation.
 Range (min … max):  3.233 ms …  3.791 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     3.298 ms              ┊ GC (median):    0.00%
 Time  (mean ± σ):   3.308 ms ± 48.426 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%        ▁▃▄▇█▅▄▃▂   ▁▁▁
  ▁▁▃▃▅▇██████████████████▇▇▇▅▆▄▅▅▄▂▃▂▂▂▂▂▂▁▂▂▂▁▂▁▂▁▂▂▂▂▁▁▂▂ ▃
  3.23 ms        Histogram: frequency by time        3.47 ms < Memory estimate: 272 bytes, allocs estimate: 11.julia> flops = n*m*(2p-1)
17175674880julia> flops / (minimum(bench.times)/1e9)
5.3131281169900205e12

For example, here we're getting over 5 TFlops of Float32 performance, which is over 10x faster than the Intel Xe Graphics G7 we had been previously using for oneAPI.jl development. At the same time, the A770 used above should be able to deliver close to 20 TFlops, so there's still room for improvement in our software stack.

To use oneAPI.jl with an Arc series GPU, you need to run Linux 6.2. At the time of writing, that kernel is still in beta, so refer to your distribution's documentation for how to install it. For example, on Arch Linux you can use the linux-mainline package from the AUR, Ubuntu has the kernel-ppa archive, Fedora provides the stable-rc repository, etc.

Other changes

• Support for Julia 1.9 has been added.

By: Tim Besard

Re-posted from: https://juliagpu.org/post/2023-02-08-oneapi_1.0/index.html

The release of oneAPI.jl 1.0 adds integration with the oneAPI Math Kernel Library (oneMKL) to accelerate linear algebra operations on Intel GPUs. It also brings support for Julia 1.9 and Intel Arc GPUs.

oneMKL integration

oneAPI.jl now uses the Intel oneAPI Math Kernel Library (oneMKL), automatically downloaded as part of oneAPI_Support_jll.jl, to accelerate a great number of BLAS and LAPACK operations on Intel GPUs. Similar to how it is implemented in our other GPU back-ends, these wrappers are available at different levels of abstraction.

At the lowest level, we use a C library that wraps the oneMKL C++ APIs. For example, the oneapi::mkl::blas::column_major::gemm function for matrix-matrix multiplication is wrapped by the C functions onemklSgemm, onemklDgemm, etc. These wrappers are used to implement low-level methods like oneMKL.gemm!:

julia> using oneAPIjulia> A = oneArray(rand(Float32, 2, 3));
2×3 oneMatrix{Float32, oneAPI.oneL0.DeviceBuffer}:
 0.44302   0.125576  0.859145
 0.674291  0.428346  0.0400119
julia> B = oneArray(rand(Float32, 3, 4))
3×4 oneMatrix{Float32, oneAPI.oneL0.DeviceBuffer}:
 0.592748   0.529413   0.0323396  0.659528
 0.22489    0.0872259  0.253291   0.376519
 0.0121506  0.591135   0.706755   0.751686
julia> C = similar(B, (2, 4));julia> oneMKL.gemm!('N', 'N', true, A, B, true, C)
2×4 oneMatrix{Float32, oneAPI.oneL0.DeviceBuffer}:
 0.301279  0.753365  0.65334   0.985274
 0.496501  0.417994  0.158581  0.63607julia> Array(C) ≈ Array(A) * Array(B)
true

Of course, these low-level functions aren't very user-friendly, so we also integrate with Julia's standard libraries where possible:

julia> A = oneArray(rand(Float32, 2, 3));
julia> B = oneArray(rand(Float32, 3, 4));julia> using LinearAlgebra
julia> C = A * B;julia> Array(C) ≈ Array(A) * Array(B)
true

The most frequently used oneMKL BLAS functions have been wrapped and integrated with Julia’s standard linear algebra libraries. If you run into a missing function, please file a request to add it, or take a look at the source and contribute to oneAPI.jl! The current state of the wrappers should make it easy to extend their functionality, as well as form a good basis for integrating with other libraries like oneDNN.

Intel Arc support

The new Arc series of discrete Intel GPUs are now fully supported by oneAPI.jl. These GPUs offer a significant performance improvement over their integrated predecessors:

julia> using oneAPI
julia> oneAPI.versioninfo()
1 device:
- Intel(R) Arc(TM) A770 Graphics [0x56a0]julia> T = Float32;
julia> n = p = m = 2048;
julia> a = oneArray(rand(T, n, p));
julia> b = oneArray(rand(T, p, m));
julia> c = oneArray(zeros(T, n, m));julia> using BenchmarkTools, LinearAlgebra
julia> bench = @benchmark oneAPI.@sync mul!(c, a, b)
BenchmarkTools.Trial: 1510 samples with 1 evaluation.
 Range (min … max):  3.233 ms …  3.791 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     3.298 ms              ┊ GC (median):    0.00%
 Time  (mean ± σ):   3.308 ms ± 48.426 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%        ▁▃▄▇█▅▄▃▂   ▁▁▁
  ▁▁▃▃▅▇██████████████████▇▇▇▅▆▄▅▅▄▂▃▂▂▂▂▂▂▁▂▂▂▁▂▁▂▁▂▂▂▂▁▁▂▂ ▃
  3.23 ms        Histogram: frequency by time        3.47 ms < Memory estimate: 272 bytes, allocs estimate: 11.julia> flops = n*m*(2p-1)
17175674880julia> flops / (minimum(bench.times)/1e9)
5.3131281169900205e12

For example, here we're getting over 5 TFlops of Float32 performance, which is over 10x faster than the Intel Xe Graphics G7 we had been previously using for oneAPI.jl development. At the same time, the A770 used above should be able to deliver close to 20 TFlops, so there's still room for improvement in our software stack.

To use oneAPI.jl with an Arc series GPU, you need to run Linux 6.2. At the time of writing, that kernel is still in beta, so refer to your distribution's documentation for how to install it. For example, on Arch Linux you can use the linux-mainline package from the AUR, Ubuntu has the kernel-ppa archive, Fedora provides the stable-rc repository, etc.

Other changes

• Support for Julia 1.9 has been added.