SDB:AMD GPGPU

Jump to: navigation, search

GPGPU on AMD (ATI) GPUs

For general GPGPU info see SDB:GPGPU.
GPGPU = General-purpose computing on graphics processing units, means using GPUs for common computations.

Warning about support level

ATI in 1990s was famous for good hardware and buggy drivers. After acquisition ATI in 2006 AMD carefully preserves this tradition. Even if AMD creates good driver - PAL OpenCL, it rapidly drops it and substitutes it with semi-working ROCm. OpenGL and Vulkan support is good due to open drivers contributed by Mesa 3D and Valve.

AMD had quit GPGPU consumer market in 2020 after dropping PAL driver. ROCm, which substitutes PAL, works on a small part of hardware, and is supported on even smaller number of GPUs. Support of GPGPU on AMD APU (iGPU) from AMD is near zero. Use another solutions if you need GPGPU. Right now these are Nvidia and Intel. But Intel consumer graphics Arc A-series lack FP64 support (link1, link2). For notebooks with AMD CPU + iGPU: if notebook has Nvidia or Intel discrete video card, then it is usable for GPGPU. Decent support level for AMD GPUs will be achieved only after maturing of Mesa's Rusticl driver. PAL driver is good. But Rusticl and PAL drivers provide OpenCL support only. You can use AMD CDNA Pro products with ROCm.

It is possible to use Vulkan drivers for computations. It works, but possibly slower than more direct approaches. Another feature: no warranties for arithmetic precision, because graphics output requires speed and can allow ignoring accuracy for results. So, Vulkan works well for AI tasks, but usage for precision computations is not recommended.

Comparison of available drivers and support level with LuxMark you can check here (might include errors).


Using CUDA code with ZLUDA and ROCm

You can use CUDA applications after installing (running) ZLUDA code translator. ZLUDA needs working ROCm installation (currently ROCm 5.7.x). Read more at https://github.com/vosen/ZLUDA and Phoronix article.
To run ROCm 5.7 on openSUSE with RDNA3 chip use environment variable 

export HSA_OVERRIDE_GFX_VERSION=10.3.0

Read more here about compatibility options.


Converting CUDA and HIP to OpenCL and Level Zero - chipStar

ChipStar is a tool for compiling and running HIP/CUDA on SPIR-V via OpenCL or Level Zero APIs. ChipStar enables porting HIP and CUDA applications to platforms which support SPIR-V as the device intermediate representation. It supports OpenCL and Level Zero as the low-level runtime alternatives. More at https://github.com/CHIP-SPV/chipStar.


Support period

Support period for AMD products from AMD is about 5 years. After 5-6 years since first release AMD drops support for its products.


Available drivers

ATI FireGL – for Terascale2 & Terascale3 chips

OpenCL 1.2 support, incompatible with modern distributions.


Mesa 3D Clover - for different hardware

OpenCL 1.2 support without image support, often hangs. Mesa 3D will deprecate it in favour of Rusticl driver.


Mesa 3D Rusticl - for different hardware

Certified OpenCL 3.0 support with image support, OpenCL 2.x features - under development, partly implemented. Available for Tumbleweed and Leap 15.6 (bug boo#1219355 was solved).
Under development. Use the newest version available with X11:XOrg repo (or devel:gcc:next:testing repo). Was introduced with Mesa 22.3. Radeonsi support (GCN1 & newer) came with Mesa 23.1, feature support (OpenCL tests conformance) - with Mesa 23.2. It works good enough since Mesa 23.3. Support with r600g (Terascale 2 & 3) needs further works.
Page: https://docs.mesa3d.org/rusticl. Supported extensions: https://mesamatrix.net/.
Rusticl from Mesa 3D 23.2 requires more than 32 GiB of RAM (works with 64 GiB RAM) for Darktable using OpenCL (link). Situation improved in next releases.


AMD ORCA – for AMD GCN1 – GCN4

OpenCL 1.2 support.
Driver 5.7.1 (23.20) is the last with ORCA OpenCL (a.k.a. "legacy OpenCL") packages.


AMD PAL – for AMD GCN2 – GCN5 and RDNA1

PAL = Platform Abstraction Library. The latest AMD driver package with PAL OpenCL inside is 20.40. Link to package is here. Newer drivers have ROCm-based OpenCL drivers for GCN5 (if not dropped) and newer chips, and ORCA drivers for GCN1 - GCN4. Works with Leap 15.2 & 15.3 (kernel 5.3), and with Leap 15.4 & 15.5 (kernel 5.14) after manual install. It might work with RDNA2 & later - not tested yet.

Pros:

  1. OpenCL 2.0/2.1 support with image support.
  2. Supports discrete GPU and iGPU (APU).
  3. Doesn't require PCIe atomics support - works with old hardware.
  4. Doesn't require DKMS.

Cons:

  1. Unsupported. Dropped by AMD in late 2020.
  2. Closed-sourced. No community support.
  3. Possibly HIP doesn't work right now with this driver.


AMD ROCm drivers – for selected AMD GCN and RDNA

OpenCL 2.x & HIP support. For discrete video cards only (unofficial support for Ryzen APUs - GCN5 iGPU and newer). For unsupported hardware use parameters or patches. ROCm packaged by AMD is available for openSUSE Leap (SUSE SLE), for Tumbleweed compile it by yourself. Some parts of ROCm are available - read here for details.


Support history:

  1. GCN3: AMD mentions community-supported ROCm.
  2. GCN4: ROCm 4.5 had dropped Polaris (GCN4, gfx803) support, so the last ROCm version for it is 4.3.1.
  3. GCN5.0: The last ROCm with support for GCN 5.0 (Vega 10, Raven, Picasso, gfx900) is 4.5.2.
  4. GCN5.1:The last ROCm series with full support for GCN 5.1 (Vega 20, Radeon VII, Renoir, Cezanne, gfx906) is 5.7.
  5. RDNA1: No official support, although might work. Try to use "export HSA_OVERRIDE_GFX_VERSION=10.3.0" parameter. Read more at SDB:AMD_GPGPU#ROCm_-_Running_on_unsupported_hardware.
  6. RDNA2: Official support with ROCm 5.x and newer.
  7. RDNA3: Support starts with ROCm 5.7.1 for Ubuntu 22.04 and Windows, for openSUSE Leap (SUSE SLE) use ROCm 6.0 and newer. You can use Distrobox to get it running.


Incompatibility of AMD ROCm with iGPU from supported architectures

ROCm (v. 6.x right now) is not supported on discrete AMD GPU if system has integrated GPU (iGPU) with architecture supported by ROCm (link).
Citation from https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/amdgpu-install.html:

Warning

ROCm doesn’t currently support integrated graphics. Should your system have an AMD IGP installed, disable it in the BIOS prior to using ROCm. If the driver can enumerate the IGP, the ROCm runtime may crash the system, even if told to omit it via HIP_VISIBLE_DEVICES.

It is rather difficult to use ROCm 6.x with {AMD GCN5.1 or AMD RDNA2 or RDNA3} iGPU + {AMD GCN5.1 or AMD RDNA2 or RDNA3} dGPU. You can disable iGPU on AM5 Ryzen 7000 desktop series. Also that means all new laptops are semi-incompatible with ROCm 6.x. Possibly it works OK with GCN5.0 iGPU - ROCm just ignores it. Otherwise we have troubles in usage ROCm 6.x with AMD iGPU + AMD dGPU on any modern laptop.
It is possible to overcome this issue by patching AMD KFD (amdkfd) driver - link1, link2. But these patches are out of tree, i.e. no support is provided by Linux kernel developers and AMD both. Other options: use virtual machine or Distrobox or Windows.
Integrated AMD GPU without discrete AMD GPU possibly works somehow.


Combining drivers

You can combine drivers. For instance, you can use 3D drivers from one source, and OpenCL - from another one. Drivers for AMD hardware consist of four parts: kernel driver (amdgpu for GCN1 & newer), OpenGL drivers, Vulkan drivers, OpenCL drivers. For Vulkan and OpenCL drivers you can install more than one set of drivers, and activate needed one by editing icd files.
Method of using OpenCL PAL AMD driver, which is described beneath, combines amdgpu + Mesa 3D drivers for OpenGL & Vulkan for Leap 15.x with OpenCL PAL AMD driver for Leap 15.2.


Sources of amdgpu driver

AMD repository for amdgpu, Mesa 3D stuff and etc. is here. Choose appropriate subfolder.
For Leap you can use amdgpu driver from AMD packages or from distribution (xf86-video-amdgpu package). To use amdgpu driver from AMD packages AMD recommends using DKMS. AMD tests ROCm with amdgpu driver + DKMS from AMD packages. Amdgpu driver from distribution works OK and is easily maintained for end user. It is better to use one source of amdgpu driver, otherwise system will hop from one to another.
With Tumbleweed you have to use amdgpu driver from distribution. It is possible to compile AMD drivers for Tumbleweed.


Sources of 3D (OpenGL & Vulkan) drivers

You can use up to four repositories for openSUSE Leap (SUSE SLE) to get video drivers:

  1. Default 3D drivers (Mesa 3D, included in distribution, Main repository)
  2. The newest drivers Mesa 3D drivers from so-called experimental X11:XOrg repository
  3. Mesa 3D drivers from AMD drivers package (https://repo.radeon.com/amdgpu/ repository, hosted by AMD)
  4. AMD proprietary drivers from AMD drivers package (https://repo.radeon.com/amdgpu/ repository, hosted by AMD)

Vulkan driver from Mesa 3D bears name RADV, Vulkan driver from AMD has name AMDVLK. Mesa 3D drivers can be faster for games, while AMD proprietary drivers might support more OpenGL/Vulkan extensions.
AMD has dropped support for GCN4 & GCN5. The latest proprietary drivers for them are 23.20 (23.30?) and 5.7.x. Open drivers work OK.


Sources of OpenCL drivers

We have two options: AMD or Mesa 3D.


Installing drivers

ATI FireGL

You may install it with downgrading X11 and other core packages. Instead of that you may use old & compatible distribution. You can use virtual machine for that.


Mesa 3D Clover

Available from Main repository. This driver hangs too much. Simply do not use it.


Mesa 3D Rusticl

Install Rusticl package. Use the latest available Mesa 3D drivers.
For Tumbleweed simply install Rusticl package.
Not available for Leap 15.5. For openSUSE Leap 15.6 add Mesa 3D experimental repository (X11:XOrg). Then install Rusticl package from it. Supposedly you will need to perform vendor change upgrade using this repo.

Enabling Rusticl

In order to use Rusticl on any platform the environment variable RUSTICL_ENABLE has to be used. Rusticl does not advertise devices for any driver by default yet as doing so can impact system stability until remaining core issues are ironed out.
To enable all radeonsi devices use 

RUSTICL_ENABLE=radeonsi

Enabling second iris and first and third radeonsi device: 

RUSTICL_ENABLE=iris:1,radeonsi:0,2


AMD ORCA

For Leap 15.5:

  1. Download amdgpu-install rpm package from https://repo.radeon.com/amdgpu-install/ into some folder.
  2. Add this folder as a repo.
  3. Install amdgpu-install package. This will add two repositories: "AMDGPU some_number repository" and "AMDGPU some_number Proprietary repository".
  4. After that you will be able to install AMD ORCA OpenCL driver by installing opencl-legacy-amdgpu-pro-icd package.
  5. Some versions of ROCm have empty Proprietary repository. To acquire packages select those versions which contain stuff.

For Leap 15.4 use installation script with "--opencl=legacy" option.
For Leap 15.3 install AMD proprietary drivers – all packages or OpenCL only (‘headless’ option). With installing headless AMD drivers you will use Mesa 3D drivers for 3D and AMD OpenCL drivers for OpenCL.
Another way: extract ORCA drivers from 20.10 drivers.
You need amdgpu kernel driver to use AMD OpenCL drivers. For GCN1 & GCN2 amdgpu driver is still in experimental phase, so you need kernel parameters to use amdgpu instead of radeon one. Add to kernel parameters “radeon.si_support=0 amdgpu.si_support=1” for GCN1, and “radeon.cik_support=0 amdgpu.cik_support=1” for GCN2. Read here for details.
With GCN1 & GCN2 and amdgpu driver VGA output in video card might not work. For GCN2 – GCN4 with AMD PAL drivers you will get OpenCL 2.0 instead of OpenCL 1.2.


AMD PAL

For Leap 15.3 use this instruction. With installing headless AMD drivers you will use Mesa 3D drivers for 3D and AMD OpenCL drivers for OpenCL.
For Leap 15.4 & 15.5 install OpenCL packages from 20.40 drivers and other packages from https://repo.radeon.com/amdgpu/ repo.
In more details for 5.7.1 repo:

  1. Download amdgpu-install rpm package into some folder.
  2. Add this folder as a repo.
  3. Install amdgpu-install package. This will add two repositories: "AMDGPU 5.7.1 repository" and "AMDGPU 5.7.1 Proprietary repository".
  4. Install these packages from AMDGPU 5.7.1 repository:
    1. amdgpu-core
    2. amdgpu-doc
    3. libdrm-amdgpu
    4. libdrm-amdgpu-common
    5. libdrm-amdgpu-devel
    6. llvm-amdgpu
    7. llvm-amdgpu-libs
    8. llvm160-amdgpu
  5. Uninstall libdrm_amdgpu1 package.
  6. Install these packages from AMDGPU 5.7.1 Pro repository:
    1. amdgpu-pro-core
    2. clinfo-amdgpu-pro
  7. Download Radeon Software for Linux 20.40 archive into some folder. Link to package is here. Extract files from tar into some folder. Look into that folder. Find RPMS subfolder in it. Add this RPMS folder with rpm packages as "AMDGPU Pro 20.40" repo.
  8. Install these packages from "AMDGPU Pro 20.40" repository:
    1. ocl-icd-amdgpu-pro
    2. ocl-icd-amdgpu-pro-devel
    3. opencl-amdgpu-pro-comgr
    4. opencl-amdgpu-pro-icd

To use clinfo from AMDGPU Pro repo run 

:~> /opt/amdgpu-pro/bin/clinfo


AMD ROCm

Documentation is here. Instructions are there.
Doesn't work with GCN5.0 - bug filed.


Foreword

ROCm is a fully open source solution from AMD for calculations on AMD GPUs using HIP and OpenCL (and CUDA via translator). AMD creates ROCm mainly for supercomputer market, for CDNA product line and its predecessor GCN5 (a.k.a. Vega graphics). Only PRO series video cards support ROCm by default, and only discrete video cards. You can use ROCm with consumer video cards with adjusted environment variable - see chapter "ROCm - Running on unsupported hardware" for details. GCN3 & GCN4 may work, possibly after recompiling of ROCm with changed parameters, GCN1 - GCN2 generations need tests & additional work. OpenCL on iGPUs with GCN5 graphics works sometimes, possibly with reduced speed (compared to PAL driver). Special edition of ROCm 3.10 (and 3.3.0) for APUs is available from Bruhnspace, but it may bear some limitations.


Requirements

Reference Guide about supported hardware is here, it is usually outdated and has mistakes.
For AMD ROCm drivers you need support for PCIe atomics feature (PCIe 3.0 and newer). Some old CPUs and motherboards do not support PCIe atomics. AMD Ryzen family (including Athlon 200GE/300GE/3000G and newer) supports PCIe atomics. PCIe bridges and re-clockers also must support PCIe atomics to provide delivering this feature between CPU(s) and GPU(s). AMD AM3+ and older sockets lacks PCIe atomics.
To check compatibility use command (with GPU being installed) 

sudo dmesg | grep -i kfd

If you find something like 

kfd kfd: added device 1002:15d8

, then your system supports PCIe atomics. Without PCIe atomics you will get something like 

kfd: skipped device 1002:7300, PCI rejects atomics


Installation

Add user to "video" and "render" groups. Install packages using AMD repositories after installing package "amdgpu-install", then add one of ROCm repository and install needed packages from it. Read AMD manual here. Some ROCm (https://repo.radeon.com/rocm/zyp/) repos (use one of them):

https://repo.radeon.com/rocm/zyp/latest/
https://repo.radeon.com/rocm/zyp/zypper/
https://repo.radeon.com/rocm/zyp/5.3.2/

Quite often you need full re-installation of ROCm packages, not just simple upgrade.
ROCm might need additional Perl packages. Adding Perl repo for Leap: 

sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/perl/$releasever/

After installation check udev rules at "/etc/udev/rules.d/"
1. for amdgpu driver (file nn-amdgpu.rules): 

KERNEL=="kfd", GROUP=="video", MODE="0660"

2. for amdkfd driver (file nn-kfd.rules): 

SUBSYSTEM=="kfd", KERNEL=="kfd", TAG+="uaccess", GROUP="video"


ROCm - Running on unsupported hardware

You can force ROCm (HIP) to run on unsupported hardware with HSA_OVERRIDE_GFX_VERSION environment variable (link 1, link 2, link 3). For instance, for RDNA1 & RDNA2 you may use 

export HSA_OVERRIDE_GFX_VERSION=10.3.0

This tricks ROCm into treating your card as a Navi 21 card (link).
For RDNA3 cards use 

export HSA_OVERRIDE_GFX_VERSION=11.0.0

It is better to use GFX number from the same generation. For GCN use GCN substitutes, not RDNA ones. To emulate Vega (GCN5) try this one (GCN5.0): 

export HSA_OVERRIDE_GFX_VERSION=9.0.0

or (GCN5.1 - works?) 

export HSA_OVERRIDE_GFX_VERSION=9.0.6

For GCN4 (Radeon RX 400 & 500 series) try to use 

ROC_ENABLE_PRE_VEGA=1

You don't need the newest ROCm to get OpenCL working with old hardware. For instance, with RX Vega 56/64 (Vega 10, GCN5.0) you can use ROCm 4.5 (4.5.2 is the latest). ROCm 5.0 needs some changes to run on GCN5.0 hardware because it uses new instructions from GCN5.1.


GPGPU on openSUSE Tumbleweed

While AMD supports HIP & OpenCL on SUSE SLE (= openSUSE Leap) only, you can use AMD OpenCL drivers with Tumbleweed by manually choosing which packages to install. Or you can compile drivers by yourself.
Of course, you can use Mesa 3D OpenCL drivers with Tumbleweed.

OpenSUSE provides some packages with GPU:ROCm repository. Add GPU:ROCm repository into your system to see all available packages.

Adding repo: 

zypper addrepo https://download.opensuse.org/repositories/science:GPU:ROCm/openSUSE_Factory/science:GPU:ROCm.repo

Installing AMD HIP runtime (https://software.opensuse.org/package/hip): 

zypper refresh
zypper install hip

Installing OpenCL driver (https://software.opensuse.org/package/amd-opencl): 

zypper refresh
zypper install amd-opencl

This will install these files: 

/usr/lib64/libamdocl64.so
/usr/share/OpenCL/vendors/amdocl64.icd
/usr/share/doc/packages/rocm-ocl-icd/LICENSE

This GPU:ROCm repo is of openSUSE-Factory kind and is incompatible with Leap. Adding this repo to openSUSE Leap will break your OS.


GPGPU on APU (iGPU)

AMD provides no official support for GPGPU on integrated GPUs for consumer products. Some ROCm versions support iGPU, but they may work much slower than expected (compared to PAL driver). Driver installer supports legacy option, but this legacy driver is ORCA driver, supports(?) GCN1 - GCN4, and has no support for GCN5 (Vega) and newer GPUs.
ROCm 6.x requires disabling iGPU to use dGPU - see previous info.
PAL driver works. But AMD drops PAL driver in favour of ROCm, so no new versions are available since 20.40 driver.
Good option is using Mesa Rusticl OpenCL driver.


Special Cases

Blender

Download archive from blender.org, extract, and use it as a portable install. Blender doesn’t support Terascale chips since 2.91, needs GCN1 and newer just to run. For Terascale 2 & 3 try using Blender 2.90 (albeit not supported, it might still work). Blender 2.x can use OpenCL. Download older Blender at Previous versions page.

You can check compatibility and speed of GPUs for Blender tasks with this site. For Blender 3.0 and newer AMD GPUs are the worst case:

  1. Blender 3.0+ doesn't support OpenCL, requires working AMD HIP from AMD ROCm. But AMD HIP is not available or doesn't work quite often. For GCN5.1 use HIP from ROCm 5.3 - 5.7. GPUs older than GCN5.1 are not officially supported with Blender + HIP.
  2. Since Blender 3.6 HIP RT is available for Windows, but not for Linux.
  3. If GPU works, then it is much slower than rivals from Nvidia and Intel.

It is possible to use ZLUDA translator to run Blender with CUDA backend (tests). With ZLUDA + CUDA + ROCm backend Blender works about 20% faster compared to HIP + ROCm.

DaVinci Resolve

DaVinci Resolve requires unconventional OpenCL extensions cl_khr_gl_sharing and cl_khr_image2d_from_buffer, which may not be available in some drivers. Extension cl_khr_image2d_from_buffer is a part of OpenCL 2.0. ORCA OpenCL driver is OpenCL 1.2 compliant, and can happily hangs with it. Use compatible drivers (PAL, ROCm/ROCr, Rusticl). Read here for details: link1, link2, link3. Also you may need to install Workstation (Proprietary) (previously AMDGPU-PRO) OpenGL drivers. OpenCL support with Rusticl is expected with Mesa 24.0 due to this and previous patches. Extension cl_khr_image2d_from_buffer is implemented in Mesa 24.0 for mesa, llvmpipe and iris, but not for radeonsi.


Hints

  1. Old AMD drivers contain installer and all packages in it. Script creates subfolder in root containing all packages and then installs packages from it. New AMD drivers installer uses AMD repositories at https://repo.radeon.com.
  2. You need special tuning to use amdgpu and radeon drivers simultaneously when system has GCN1/GCN2 GPUs (link1, link2).
  3. Installation of ROCm drivers blacklists "radeon" video driver by default.
  4. Terascale chips lack some commands, so you need tuning to use OpenCL in Darktable via settings in a file - read Darktable manual for details.
  5. GPGPU on Terascale 1 and older AMD/ATI chips is not supported (well, it is possible, but meaningless).
  6. AMD officially supports GPGPU only with selected discrete video cards and server APUs. However, GPGPU might work on consumer APUs.
Retrieved from "https://en.opensuse.org/index.php?title=SDB:AMD_GPGPU&oldid=185862"