SDB:AMD GPGPU

Jump to: navigation, search

GPGPU on AMD (ATI) GPUs

For general GPGPU info see SDB:GPGPU.
GPGPU = General-purpose computing on graphics processing units, means using GPUs for common computations.

Warning about support level

ATI in 1990s was famous for good hardware and buggy drivers. After acquisition ATI in 2006 AMD carefully preserves this tradition. Even if AMD creates good driver - PAL OpenCL, it rapidly drops it and substitutes it with semi-working ROCm. OpenGL and Vulkan support is good due to open drivers contributed by Mesa 3D and Valve.

AMD had quit GPGPU consumer market in 2020 after dropping PAL driver. ROCm, which substitutes PAL, works on a small part of hardware, and is supported on even smaller number of GPUs. Support of GPGPU on AMD APU (iGPU) from AMD is near zero. Use another solutions if you need GPGPU. Right now these are Nvidia and Intel. But Intel consumer graphics Arc A-series lack FP64 support (link1, link2). For notebooks with AMD CPU + iGPU: if notebook has Nvidia or Intel discrete video card, then it is usable for GPGPU. Decent support level for AMD GPUs will be achieved only after maturing of Mesa's Rusticl driver. PAL driver is good. But Rusticl and PAL drivers provide OpenCL support only. You can use AMD CDNA Pro products with ROCm.

Comparison of available drivers and support level with LuxMark you can check here (may include errors).


Support period

Support period for AMD products from AMD is about 5 years. After 5-6 years since first release AMD drops support for its products.


Available drivers


ATI FireGL – for Terascale 2 & 3 chips

OpenCL 1.2 support, incompatible with modern distributions.

Mesa 3D Clover - for different hardware

OpenCL 1.2 support without image support, often hangs. Mesa 3D will deprecate it in favour of Rusticl driver.

Mesa 3D Rusticl - for different hardware.

Under development. May not work right now with your hardware. Was introduced with Mesa 22.3. Radeonsi support (GCN1 & newer) came with Mesa 23.1, feature support (OpenCL tests conformance) - with Mesa 23.2. Support with r600g (Terascale 2 & 3) needs further tests.
Page: https://docs.mesa3d.org/rusticl
Certified OpenCL 3.0 support with image support, OpenCL 2.x features - under development.

AMD ORCA – for AMD GCN 1 – GCN 4.

OpenCL 1.2 support.

AMD PAL – for AMD GCN 2 – GCN 5 and RDNA 1.

PAL = Platform Abstraction Library. The latest AMD driver package with PAL OpenCL inside is 20.40. Link to package is here. Newer drivers have ROCm-based OpenCL drivers for GCN5 and newer chips, and ORCA drivers for GCN1 - GCN4. Works with Leap 15.2 & 15.3 (kernel 5.3), and with Leap 15.4 & 15.5 (kernel 5.14) after manual install. May work with RDNA2 & later.

Pros:

  1. OpenCL 2.0/2.1 support with image support.
  2. Supports discrete GPU and iGPU (APU).
  3. Doesn't require PCIe atomics support - works with old hardware.
  4. Doesn't require DKMS.

Cons:

  1. Unsupported. Dropped by AMD in late 2020.
  2. Closed-sourced. No community support.
  3. HIP doesn't work right now with this driver.

AMD ROCm drivers – for selected AMD GCN and RDNA.

OpenCL 2.x & HIP support. For discrete video cards only (unofficial support for Ryzen APUs - GCN5 iGPU and newer). For unsupported hardware patches are needed.


Support history:

  1. GCN 3: AMD mentions community supported ROCm.
  2. GCN 4: ROCm 4.5 had dropped Polaris (GCN4, gfx803) support, so the last ROCm version for it is 4.3.1.
  3. GCN 5: The last ROCm with support for GCN 5.0 (Vega 10, Raven, Picasso, gfx900) is 4.5.2.
  4. GCN 5.1:The last ROCm series with full support for GCN 5.1 (Vega 20, Radeon VII, Renoir, Cezanne, gfx906) is 5.7.
  5. RDNA 1: No official support, although might work. Try to use "export HSA_OVERRIDE_GFX_VERSION=10.3.0" parameter. Read more at SDB:AMD_GPGPU#ROCm_-_Running_on_unsupported_hardware.
  6. RDNA 2: Official support with ROCm 5.x and newer.
  7. RDNA 3: Support starts with ROCm 5.7.1 for Ubuntu 22.04 and Windows, for SUSE (openSUSE) it is expected with ROCm 6.0. You can use Distrobox to get it running.


Combining drivers

You can combine drivers. For instance, you can use 3D drivers from one source, and OpenCL - from another one. Drivers for AMD hardware consist of four parts: kernel driver (amdgpu for GCN 1 & newer), OpenGL drivers, Vulkan drivers, OpenCL drivers. For Vulkan and OpenCL drivers you can install more than one set of drivers, and activate needed one by editing icd files.
Method of using OpenCL PAL AMD driver, which is described beneath, combines amdgpu + Mesa 3D drivers for OpenGL & Vulkan for Leap 15.x with OpenCL PAL AMD driver for Leap 15.2.

Sources of amdgpu driver

AMD repository for amdgpu, Mesa 3D stuff and etc. is here. Choose appropriate subfolder.
For Leap you can use amdgpu driver from AMD packages or from distribution (xf86-video-amdgpu package). To use amdgpu driver from AMD packages AMD recommends using DKMS. AMD tests ROCm with amdgpu driver + DKMS from AMD packages. Amdgpu driver from distribution works OK and is easily maintained for end user. It is better to use one source of amdgpu driver, otherwise system will hop from one to another.
With Tumbleweed you have to use amdgpu driver from distribution.

Sources of 3D (OpenGL & Vulkan) drivers

You can use up to four repositories for openSUSE Leap (SUSE SLE) to get video drivers:

  1. Default 3D drivers (Mesa 3D, included in distribution, Main repository)
  2. The newest drivers Mesa 3D drivers from so-called experimental X11:XOrg repository
  3. Mesa 3D drivers from AMD drivers package (https://repo.radeon.com/amdgpu/ repository, hosted by AMD)
  4. AMD proprietary drivers from AMD drivers package (https://repo.radeon.com/amdgpu/ repository, hosted by AMD)

Vulkan driver from Mesa 3D bears name RADV, Vulkan driver from AMD has name AMDVLK. Mesa 3D drivers can be faster for games, while AMD proprietary drivers might support more OpenGL/Vulkan extensions.
AMD has dropped support for GCN4 & GCN5. The latest proprietary drivers for them are 23.20 (23.30?) and 5.7.x. Open drivers work OK.

Sources of OpenCL drivers

We have two options: AMD or Mesa 3D.


Installing drivers


ATI FireGL

You may install it with downgrading X11 and other core packages. Instead of that you may use old & compatible distribution. You can use virtual machine for that.

Mesa 3D Clover

Available from Main repository. This driver hangs too much. Simply do not use it.

Mesa 3D Rusticl

Install Rusticl package. Use the latest available Mesa 3D drivers.
For Tumbleweed simply install Rusticl package.
(Not available right now for Leap 15.4 & 15.5). For openSUSE Leap 15.5 add Mesa 3D experimental repository (X11:XOrg). Then install Rusticl package from it. Supposedly you will need to perform vendor change upgrade using this repo.

AMD ORCA

For Leap 15.5:

  1. Download amdgpu-install rpm package from https://repo.radeon.com/amdgpu-install/ into some folder.
  2. Add this folder as a repo.
  3. Install amdgpu-install package. This will add two repositories: "AMDGPU some_number repository" and "AMDGPU some_number Proprietary repository".
  4. After that you will be able to install AMD ORCA OpenCL driver by installing opencl-legacy-amdgpu-pro-icd package.
  5. Some versions of ROCm have empty Proprietary repository. To acquire packages select those versions which contain stuff.

For Leap 15.4 use installation script with "--opencl=legacy" option.
For Leap 15.3 install AMD proprietary drivers – all packages or OpenCL only (‘headless’ option). With installing headless AMD drivers you will use Mesa 3D drivers for 3D and AMD OpenCL drivers for OpenCL.
Another way: extract ORCA drivers from 20.10 drivers.
You need amdgpu kernel driver to use AMD OpenCL drivers. For GCN 1 & 2 amdgpu driver is still in experimental phase, so you need kernel parameters to use amdgpu instead of radeon one. Add to kernel parameters “radeon.si_support=0 amdgpu.si_support=1” for GCN1, and “radeon.cik_support=0 amdgpu.cik_support=1” for GCN2. Read here for details.
With GCN 1 & 2 and amdgpu driver VGA output in video card might not work. For GCN 2 – 4 with AMD PAL drivers you will get OpenCL 2.0 instead of OpenCL 1.2.

AMD PAL

For Leap 15.3 use this instruction. With installing headless AMD drivers you will use Mesa 3D drivers for 3D and AMD OpenCL drivers for OpenCL.
For Leap 15.4 & 15.5 install OpenCL packages from 20.40 drivers and other packages from https://repo.radeon.com/amdgpu/ repo.
In more details for 5.7.1 repo:

  1. Download amdgpu-install rpm package into some folder.
  2. Add this folder as a repo.
  3. Install amdgpu-install package. This will add two repositories: "AMDGPU 5.7.1 repository" and "AMDGPU 5.7.1 Proprietary repository".
  4. Install these packages from AMDGPU 5.7.1 repository:
    1. amdgpu-core
    2. amdgpu-doc
    3. libdrm-amdgpu
    4. libdrm-amdgpu-common
    5. libdrm-amdgpu-devel
    6. llvm-amdgpu
    7. llvm-amdgpu-libs
    8. llvm160-amdgpu
  5. Uninstall libdrm_amdgpu1 package.
  6. Install these packages from AMDGPU 5.7.1 Pro repository:
    1. amdgpu-pro-core
    2. clinfo-amdgpu-pro
  7. Download Radeon Software for Linux 20.40 archive into some folder. Link to package is here. Extract files from tar into some folder. Look into that folder. Find RPMS subfolder in it. Add this RPMS folder with rpm packages as "AMDGPU Pro 20.40" repo.
  8. Install these packages from "AMDGPU Pro 20.40" repository:
    1. ocl-icd-amdgpu-pro
    2. ocl-icd-amdgpu-pro-devel
    3. opencl-amdgpu-pro-comgr
    4. opencl-amdgpu-pro-icd

To use clinfo from AMDGPU Pro repo run 

:~> /opt/amdgpu-pro/bin/clinfo


AMD ROCm

Documentation is here.
Doesn't work with GCN 5.0 - bug filed.

Foreword

ROCm is a fully open source solution from AMD for calculations on AMD GPUs using HIP and OpenCL (and CUDA via translator). AMD creates ROCm mainly for supercomputer market, for CDNA product line and its predecessor GCN5 (a.k.a. Vega graphics). Only PRO series video cards support ROCm by default, and only discrete video cards. You can use ROCm with consumer video cards with adjusted environment variable - see chapter "ROCm - Running on unsupported hardware" for details. GCN3 & GCN4 may work, possibly after recompiling of ROCm with changed parameters, GCN1 - GCN2 generations need tests & additional work. OpenCL on iGPUs with GCN5 graphics works sometimes, possibly with reduced speed (compared to PAL driver). Special edition of ROCm 3.10 (and 3.3.0) for APUs is available from Bruhnspace, but it may bear some limitations.

Requirements

Reference Guide about supported hardware is here, it is usually outdated and has mistakes.
For AMD ROCm drivers you need support for PCIe atomics feature (PCIe 3.0 and newer). Some old CPUs and motherboards do not support PCIe atomics. AMD Ryzen family (including Athlon 200GE/300GE/3000G and newer) supports PCIe atomics. PCIe bridges and re-clockers also must support PCIe atomics to provide delivering this feature between CPU(s) and GPU(s). AMD AM3+ and older sockets lacks PCIe atomics.
To check compatibility use command (with GPU being installed) 

sudo dmesg | grep -i kfd

If you find something like 

kfd kfd: added device 1002:15d8

, then your system supports PCIe atomics. Without PCIe atomics you will get something like 

kfd: skipped device 1002:7300, PCI rejects atomics

Installation

Add user to "video" and "render" groups. Install packages using AMD repositories after installing package "amdgpu-install", then add one of ROCm repository and install needed packages from it. Read AMD manual here. Some ROCm (https://repo.radeon.com/rocm/zyp/) repos (use one of them):

https://repo.radeon.com/rocm/zyp/latest/
https://repo.radeon.com/rocm/zyp/zypper/
https://repo.radeon.com/rocm/zyp/5.3.2/

Sometimes you need full re-installation of ROCm packages, not just simple upgrade.

ROCm - Running on unsupported hardware

You can force ROCm (HIP) to run on unsupported hardware with HSA_OVERRIDE_GFX_VERSION environment variable (link 1, link 2, link 3). For instance, you may use 

export HSA_OVERRIDE_GFX_VERSION=10.3.0

This tricks ROCm into treating your card as a Navi 21 card (link). It is better to use GFX number from the same generation. For GCN use GCN substitutes, not RDNA ones. To emulate Vega (GCN5) try this one: 

export HSA_OVERRIDE_GFX_VERSION=9.0.0

or 

export HSA_OVERRIDE_GFX_VERSION=9.0.6


GPGPU on openSUSE Tumbleweed

While AMD supports HIP & OpenCL on SUSE SLE (= openSUSE Leap) only, you can use AMD OpenCL drivers with Tumbleweed by manually choosing which packages to install. Or you can compile drivers by yourself.
Of course, you can use Mesa 3D OpenCL drivers with Tumbleweed.


GPGPU on APU (iGPU)

AMD provides no official support for GPGPU on integrated GPUs for consumer products. Some ROCm versions support iGPU, but they may work much slower than expected (compared to PAL driver). Driver installer supports legacy option, but this legacy driver is ORCA driver, supports(?) GCN1 - GCN4, and has no support for GCN5 (Vega) and newer GPUs.
PAL driver works. But AMD drops PAL driver in favour of ROCm, so no new versions are available since 20.40 driver.
Good option is using Mesa Rusticl OpenCL driver.


Special Cases

Blender

Download archive from blender.org, extract, and use it as a portable install. Blender doesn’t support Terascale chips since 2.91, needs GCN 1 and newer just to run. For Terascale 2 & 3 try using Blender 2.90 (albeit not supported, it might still work). Blender 2.x can use OpenCL. Download older Blender at Previous versions.

You can check compatibility and speed of GPUs for Blender tasks with this site. For Blender 3.0 and newer AMD GPUs are the worst case:

  1. Blender 3.0+ doesn't support OpenCL, emits CUDA commands, requires working AMD HIP from AMD ROCm. But AMD HIP is not available or doesn't work quite often. For GCN 5.1 use HIP from ROCm 5.3 - 5.7. GPUs older than GCN 5.1 are not officially supported with Blender + HIP.
  2. Since Blender 3.6 HIP RT is available for Windows, but not for Linux.
  3. RDNA 3 is supported since ROCm 5.7.1, which is available for Windows and Ubuntu, but not for SUSE/openSUSE.
  4. If GPU works, then it is much slower than rivals from Nvidia and Intel.
  5. Blender works faster on Linux compared to Windows with Nvidia and Intel GPUs, but slower with AMD HIP.

DaVinci Resolve

DaVinci Resolve requires unconventional OpenCL extensions cl_khr_gl_sharing and cl_khr_image2d_from_buffer, which may not be available in some drivers. Extension cl_khr_image2d_from_buffer is a part of OpenCL 2.0. ORCA OpenCL driver is OpenCL 1.2 compliant, and can happily hangs with it. Use compatible drivers (PAL, ROCm/ROCr, Rusticl). Read here for details: link1, link2, link3. Also you may need to install Workstation (Proprietary) (previously AMDGPU-PRO) OpenGL drivers. OpenCL support with Rusticl is expected with Mesa 24.0 due to this and previous patches.


Hints

  1. Old AMD drivers contain installer and all packages in it. Script creates subfolder in root containing all packages and then installs packages from it. New AMD drivers installer uses AMD repositories at https://repo.radeon.com.
  2. You need special tuning to use amdgpu and radeon drivers simultaneously when system has GCN1/GCN2 GPUs.
  3. Installation of ROCm drivers blacklists "radeon" video driver by default.
  4. Terascale chips lack some commands, so you need tuning to use OpenCL in Darktable via settings in a file - read Darktable manual for details.
  5. GPGPU on Terascale 1 and older AMD/ATI chips is not supported (well, it is possible, but meaningless).
  6. AMD officially supports GPGPU only with selected discrete video cards and server APUs. However, GPGPU might work on consumer APUs.
Retrieved from "https://en.opensuse.org/index.php?title=SDB:AMD_GPGPU&oldid=181643"