Pluggable CPU schedulers

Jump to: navigation, search

Getting started on Tumbleweed

The following should get you to install a kernel containing the "sched-ext" patch and the required toolchain to run custom CPU schedulers. In order to test that everything is set up correctly, we'll compile and run a scheduler from the repository of userspace sched-ext tools.

These directions have been tested on x86_64 only.

Enabled kernel

Adding the repository with the patched kernel: 

# SCX_REPO_URI=https://download.opensuse.org/repositories/home:/ggherdovich:/sched-ext-4:/kernel-scx/standard
# zypper ar $SCX_REPO_URI scx
# zypper ref

Searching for "kernel-default" in that repo: 

# zypper se --details --repo scx --type package /^kernel-default$/

S  | Name           | Type    | Version               | Arch   | Repository
---+----------------+---------+-----------------------+--------+-----------
i+ | kernel-default | package | 6.10~rc2-1.1.g12821a0 | x86_64 | scx

Eventually this package will be renamed to "kernel-scx". For the moment, it's "kernel-default" like the standard kernel, so we need to identify it by its version and git commit hash.

Installing the kernel and kernel headers, necessary to build BPF programs: 

# zypper install kernel-default-6.10~rc2-1.1.g12821a0.x86_64
# zypper install kernel-default-devel-6.10~rc2-1.1.g12821a0.x86_64

Development toolchain

Essentially LLVM, the Rust compiler and libbpf. 

# zypper in meson cmake-full
# zypper in clang18 llvm18
# zypper in libbpf1 libbpf-devel bpftool
# zypper in rustup

Your first custom scheduler

We'll clone the repository of userspace sched-ext tooling and sample schedulers, and build the entire thing: 

$ git clone https://github.com/sched-ext/scx.git
$ cd scx
$ sudo meson setup build --prefix $HOME -D libbpf_a=disabled -D bpftool=disabled

A few things to note here:

  • The option --prefix $HOME makes so that subsequent invocations of meson install place all binaries (the schedulers we're building) at $HOME/bin instead of the default /usr/local/bin. This isn't necessary, but shows how to control the installation directory, which is good to know.
  • The variable libbpf_a controls static linkage to libbpf. We're setting it to disabled, as we're opting for dynamic linkage; we've installed the shared library with the libbpf1 package, and the header files with the libbpf-devel package (see the previous section).
  • The variable bpftool controls which bpftool(8) we're using. With the value disabled, we're going for the system one, installed via the bpftool package.
  • sudo(8) is necessary as bpftool(8) is a system binary and is stored at /sbin; it requires root. This is a little awkward, as now all build artifacts will be owned by root, so the following meson(1) invocations will require sudo(8) as well (meson compile, meson install). We'll take it for now. At least the value of $HOME is resolved by the shell before invoking sudo(8), so the binaries will end up at the right place.

We're now ready for compilation and installation: 

$ sudo meson compile -C build
$ sudo meson install -C build

Be aware that in case anything goes wrong and you need to restart the build process, all is needed for cleaning up is the deletion of the build directory: 

$ rm -rf build

If the build is successful, our freshly built schedulers will be placed at $HOME/bin: 

$ ls -1 $HOME/bin
scx_bpfland
scx_central
scx_flatcg
scx_lavd
scx_layered
scx_mitosis
scx_nest
scx_pair
scx_qmap
scx_rlfifo
scx_rustland
scx_rusty
scx_simple
scx_userland

There's quite a lot of these! They can actually be compiled separately, in case you're interested in just one, or a few. Meson(1) has an introspect command that can be used to list build targets. It will output in JSON format; by feeding that data into a jq(1) query, we can list the interesting targets like so: 

$ meson introspect --targets build \
      | jq 'map(select(.type == "executable" or .type == "custom")) | .[] | .name'
"cc_cflags_probe"
"libbpf"
"bpftool_target"
"scx_utils"
"scx_rustland_core"
"scx_layered"
"scx_mitosis"
"scx_rusty"
"scx_rustland"
"scx_rlfifo"
"scx_bpfland"
"scx_lavd"
"rust_scheds"
"scx_simple"
"scx_qmap"
"scx_central"
"scx_pair"
"scx_flatcg"
"scx_userland"
"scx_nest"

For example, to build only the "scx_flatcg" scheduler, the commands are: 

$ sudo meson compile -C build scx_flatcg
$ sudo meson install -C build

Anyways, we're now ready to load some of our new schedulers. I hear scx_rustland is a good one: 

$ sudo $HOME/bin/scx_rustland
12:35:19 [INFO] libbpf: struct_ops rustland: member cgroup_init not found in kernel, skipping it as it's set to zero
12:35:19 [INFO] libbpf: struct_ops rustland: member cgroup_exit not found in kernel, skipping it as it's set to zero
12:35:19 [INFO] libbpf: struct_ops rustland: member cgroup_prep_move not found in kernel, skipping it as it's set to zero
12:35:19 [INFO] libbpf: struct_ops rustland: member cgroup_move not found in kernel, skipping it as it's set to zero
12:35:19 [INFO] libbpf: struct_ops rustland: member cgroup_cancel_move not found in kernel, skipping it as it's set to zero
12:35:19 [INFO] libbpf: struct_ops rustland: member cgroup_set_weight not found in kernel, skipping it as it's set to zero

12:35:19 [INFO] RustLand scheduler attached - 4 CPUs

12:35:20 [INFO] min_vruntime=24297415 max_vruntime=24297415 delta=0us slice=5000us
12:35:20 [INFO]   tasks=6
12:35:20 [INFO]   nr_user_dispatches=13 nr_kernel_dispatches=5
12:35:20 [INFO]   nr_cancel_dispatches=0 nr_bounce_dispatches=0
12:35:20 [INFO]   nr_running=1 nr_waiting=0 [nr_queued=0 + nr_scheduled=0]
12:35:20 [INFO]   nr_failed_dispatches=0 nr_sched_congested=0 nr_page_faults=0 [OK]
12:35:20 [INFO] Running tasks:
12:35:20 [INFO]   core  0 cpu  0 pid=0
12:35:20 [INFO]   core  1 cpu  1 pid=0
12:35:20 [INFO]   core  2 cpu  2 pid=[self]
12:35:20 [INFO]   core  3 cpu  3 pid=0

12:35:21 [INFO] min_vruntime=24297415 max_vruntime=24297415 delta=0us slice=5000us
12:35:21 [INFO]   tasks=6
12:35:21 [INFO]   nr_user_dispatches=13 nr_kernel_dispatches=23
12:35:21 [INFO]   nr_cancel_dispatches=0 nr_bounce_dispatches=0
12:35:21 [INFO]   nr_running=1 nr_waiting=0 [nr_queued=0 + nr_scheduled=0]
12:35:21 [INFO]   nr_failed_dispatches=0 nr_sched_congested=0 nr_page_faults=0 [OK]
12:35:21 [INFO] Running tasks:
12:35:21 [INFO]   core  0 cpu  0 pid=0
12:35:21 [INFO]   core  1 cpu  1 pid=[self]
12:35:21 [INFO]   core  2 cpu  2 pid=500
12:35:21 [INFO]   core  3 cpu  3 pid=0

12:35:22 [INFO] min_vruntime=24297415 max_vruntime=24297415 delta=0us slice=5000us
12:35:22 [INFO]   tasks=6
12:35:22 [INFO]   nr_user_dispatches=13 nr_kernel_dispatches=38
12:35:22 [INFO]   nr_cancel_dispatches=0 nr_bounce_dispatches=0
12:35:22 [INFO]   nr_running=0 nr_waiting=0 [nr_queued=0 + nr_scheduled=0]
12:35:22 [INFO]   nr_failed_dispatches=0 nr_sched_congested=0 nr_page_faults=0 [OK]
12:35:22 [INFO] Running tasks:
12:35:22 [INFO]   core  0 cpu  0 pid=0
12:35:22 [INFO]   core  1 cpu  1 pid=[self]
12:35:22 [INFO]   core  2 cpu  2 pid=0
12:35:22 [INFO]   core  3 cpu  3 pid=0

12:35:23 [INFO] min_vruntime=24297415 max_vruntime=24297415 delta=0us slice=5000us
12:35:23 [INFO]   tasks=5
12:35:23 [INFO]   nr_user_dispatches=13 nr_kernel_dispatches=71
12:35:23 [INFO]   nr_cancel_dispatches=0 nr_bounce_dispatches=0
12:35:23 [INFO]   nr_running=0 nr_waiting=0 [nr_queued=0 + nr_scheduled=0]
12:35:23 [INFO]   nr_failed_dispatches=0 nr_sched_congested=0 nr_page_faults=0 [OK]
12:35:23 [INFO] Running tasks:
12:35:23 [INFO]   core  0 cpu  0 pid=1498
12:35:23 [INFO]   core  1 cpu  1 pid=[self]
12:35:23 [INFO]   core  2 cpu  2 pid=0
12:35:23 [INFO]   core  3 cpu  3 pid=0
^C

EXIT: Scheduler unregistered from user space
12:35:24 [INFO] Unregister RustLand scheduler
Retrieved from "https://en.opensuse.org/index.php?title=Pluggable_CPU_schedulers&oldid=187898"