Full Disk Encryption (FDE)

MicroOS and Tumbleweed have now images using full disk encryption based on systemd, for bootloaders that follows the boot loader specification (BLS[1]).

systemd-boot bootloader is well know for introducing this specification, were the bootloader entries are not small configuration files in the EFI partition (type#1), or more recently PE bundles that agglutinate the kernel, the initrd and some metadata like the command line in a single file known as unified kernel image or UKI (type#2)

Recently openSUSE backported from Fedora the patches that will enable BLS type#1 entries in GRUB2. Other bootloaders like zipl (s390x) or Petitboot (PowerPC) has also support for the BLS, and because systemd-boot and GRUB2 are supported in arm64, BLS cover all the architectures that we care of.

The new MicroOS[2] and Tumbleweed[3] images propose a full disk encryption architecture based on systemd that depends on having a bootloader that follows the BLS.

With systemd we can enroll traditional passwords, together with automatic unlock based on TPM2 or FIDO2 keys. In the case of TPM2, the automatic unlock will happens only in the systems is in a good known state (healthy state), determined by measure boot[4] that includes all the firmware, secure boot certificates, the bootloader, kernel, initrd and command line. This means that is something in the boot chain has been altered (for example, if the initrd is not the one that we used during the installation), the TPM2 will detect the change and will refuse to deliver the password to unlock the LUKS2 device, and a password will be asked.

Installation

The images are deployed as qcow2 images for KVM. They can be run in QEMU via libvirt, for example. The most direct way of running them is using virt-manager or virt-install to create a machine with UEFI and TPM2.

First, download the sdboot image from here [8]. The simplest command to run the image using libvirt is the following:

 virt-install --name microos-sdboot \
   --ram 2048 \
   --boot uefi \
   --tpm backend.type=emulator,backend.version=2.0,model=tpm-tis \
   --import --file ./openSUSE-MicroOS.x86_64-kvm-and-xen-sdboot.qcow2 \
   --osinfo opensuse-factory

Notice the --boot uefi option to enable UEFI and the -tpm flag to enable an emulated software tpm. virt-viewer will be started automatically while the virtual machine is booting up. During the first boot the systems will ask if you want encrypt the current device, and at the end of the first boot, a jeos-firstboot will proceed with the enrollment.

The system generated a long recovery key. This key needs to be kept safe (scan the QR code with a phone for a quick access to the key). This key is used to access the LUKS2 rootfs device.

A menu will give us the option of enrolling the TPM2 (if one is detected) with or without an extra PIN. With the TPM2 we can automatically unlock the device if the system is in good state (see measured boot from [4]). If the PIN as enrolled, it will be asked after the validation of the system. If both matched the key to unlock the LUKS2 key device will be released and the boot process will continue.

If during the enrollment process a FIDO2 key is detected, the menu will give the option to enroll it. Not all the FIDO2 keys can be enrolled, they need to support the hmac-secret extension. To see if out key support it, one option is to see if it can be listed:

 systemd-cryptenroll --fido2-device=auto "$DEV"

Optionally, we can enroll the root password or a new one.

The recovery key is also used as a recovery PIN in case that we enrolled a TPM2 with systemd-pcrlock (default option). Do not confuse the recovery PIN with the one created for the TPM2 + PIN enrollment option. The recovery PIN will be used when the prediction fails and we need to register a new prediction, without requiring a the tedious process of re-enrollment of a new prediction from scratch, something that will require a modification of the LUKS2 header of all of the encrypted devices.

Now the system can be operated normally. After an update that changes some component of the boot chain, new predictions for the TPM2 will be generated so the next reboot will still considered the system as a healthy one and no password the open the LUKS2 device will be asked.

You can check that the system is using full disk encryption by running the following command (replace /dev/vda3 with your disk, if different):

 systemd-cryptenroll /dev/vda3

The command will show all the keyslots, including the fido2 and tpm if everything is working correctly.

Installation with YaST

Because today YaST does not support this kind of installation, the full process will involve some manual steps.

We first needs to download the MicroOS DVD image[7]. Remember that for a correct installation we need a UEFI system with TPM2.

Start the normal installation, and once the general Installation Settings appears select Partitioning. From there go to Expert Partitioner / Start with Current Proposal. In this windows remove the two GRUB2 subvolumes: @/boot/grub2/i386-pc, and @/boot/grub2/x86_64-efi. You will see two partitions, one that contains the rootfs and another one for /var. We need to encrypt both partitions. Select one, press the Edit.. button and enable the Encrypt Device checkbox. In the next windows the selected Encryption method should be Regular LUKS2. Fill the encryption password and in the Password-Based Key Derivation Function select Argon2id.

Be sure to encrypt both partitions. You will be welcomed by a lot of warning popups. They are about that GRUB2 does not support this kind of encryption. But for this installation we will use later systemd-boot.

Once again in the generic Installation Settings windows, go to the software section. From there change the view and install this list of packages: tmp2.0-tools, tpm2-0-tss, libtss2-tcti-device0, libtss2-esys0, libtss2-fapi-common, libtss2-fapi1, libtss2-rc0. Eventually YaST will install this, maybe via a new pattern.

One last time we will go to the Installation Settings view, and from there we select the Booting section. Select Systemd boot from the Boot Loader' drop down and optionally adjust the command line parameters for your system. For example, because I am doing an installation in a local VM I decided to turn off some of the security options.

Complete the installation. A very worrisome message will appears at the end, indicating problems during the initrd creation. This fix is WIP, but ignore it for now.

Reboot the system and check that indeed systemd-boot is showing the main menu. Enter the LUKS2 password to continue the boot.

Once in the terminal check that all your partitions are encrypted. For example, in my VM:

 localhost:~ # lsblk
 NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
 sr0          11:0    1 1024M  0 rom   
 vda         253:0    0   20G  0 disk  
 ├─vda1      253:1    0  512M  0 part  /boot/efi
 ├─vda2      253:2    0  8.2G  0 part  
 │ └─cr_root 254:1    0  8.2G  0 crypt /usr/local
 │                                     /srv
 │                                     /opt
 │                                     /home
 │                                     /boot/writable
 │                                     /.snapshots
 │                                     /root
 │                                     /
 └─vda3      253:3    0 11.3G  0 part  
   └─cr_var  254:0    0 11.3G  0 crypt /var

 localhost:~ # systemd-cryptenroll /dev/vda2
 SLOT TYPE    
    0 password
 localhost:~ # systemd-cryptenroll /dev/vda3
 SLOT TYPE    
    0 password

For the automatic unlock, we are going to use the TPM2, for that we need first to edit /etc/crypttab and add tmp2-device=auto for all the entries. In my case I would have something like:

 cr_root  UUID=4ffba63a-1a7a-408f-9d2e-0f9a49390559  none  x-initrd.attach,tmp2-device=auto
 cr_var   UUID=5aecbaca-5103-44a2-81ca-a502d619b2d7  none  tmp2-device=auto

Now we need to enumerate the PCR that we are going to follow.

 echo "FDE_SEAL_PCR_LIST=0,2,4,7,9" > /etc/sysconfig/fde-tools

In a future version of this guide all this process will be done automatically and a recovery password will be generated like the one done for the image based MicroOS. But for now the last steps are to generate a new initrd that will include the /etc/crypttab changes and generate the new predictions for those PCRs. This can be done with a single command:

 sdbootutil mkinitrd

Finally enroll both devices:

 systemd-cryptenroll \
   --tpm2-device=auto \
   --tpm2-pcrlock=/var/lib/systemd/pcrlock.json \
   /dev/vda2

 systemd-cryptenroll \
   --tpm2-device=auto \
   --tpm2-pcrlock=/var/lib/systemd/pcrlock.json \
   /dev/vda3

How it works

A somehow detailed description of how this work internally can be found on the openSUSE wiki[5] and in the initial announcement[6].

Here it is the short version.

As commented measured boot[4] is a process that will asses the health of the systems. When the machine is turn on, after some initialization a component of the firmware will check if there is a TPM2 device available. This multifaceted device can be used as a "root of trust" of some measurements that can be done in the system.

This means that the firmware can calculate the hash of the next component of the boot process, and extend internal registers inside the TPM2, known as PCR', to register the calculated hash in a cryptographic way. This extension operation is just the append of the calculated hash with the current register value, and calculate again the hash of the compound. This calculated hash will be the new value of the PCR.

This operation makes impossible to write a specific value in one of those register, and the final value can only be reproduced replicating the sequence of values used to extend the register. This means that if for some reason one of those values is different, the final value of the PCR will be something very different at the expected value.

This is the core of measured boot. Each component of the boot chain (the multiple steps in the UEFI firmware, the shim bootloader, the bootloader itself, the kernel and initrd) will load the next step in the chain, calculate the hash of the memory blob (or data), and extend one of those PCRs following a public specification. The final values of the registers inside the TPM2 represent the status of the system.

The TPM2 can be used to encrypt (seal) some data, like for example a password, based on the expected value of those register is such a way that if the registers match some values then the TPM2 can decrypt (unseal in the jargon). This means that we can use the TPM2 to store the key of a LUKS2 device, and this key will be delivered back to the system only in case that this system is in a good state.

That is the basis of a FDE without requesting the password after every boot, and systemd with systemd-cryptenroll and other tools promise to deliver. The recommended systemd approach requires the use of read-only devices via dm-verity and UKIs for the kernels. But the openSUSE MicroOS and Tumbleweed project created a set of tools that this same architecture instantiated for this distribution, but adapted to the current status (separated kernel and initrd, transactions based on btrfs snapshots, etc)

The new tools used are:

disk-encryption-tool
sdbootutil
dracut-pcr-signature

disk-encryption-tool is present only on image based distributions, and will take care of encrypting the image on the first boot. Doing that will guarantee that a new master key will be generated for each system, and no shared LUKS2 key slot with some recovery password will be stored in the image. This tool also will use jeos-firstboot to make the initial enrollment, using systemd-cryptenroll and the next tool, sdbootutil.

The management of boot entries in the traditional GRUB2 are very different from the ones used in bootloaders that follows the BLS. When a new snapshot is generated, the new kernel needs to be copied from the rootfs into a specific place in the ESP (the FAT32 partition required by UEFI that in openSUSE is in '/boot/efi'). A new initrd needs to be regenerated if required, or an old one needs to be reused. And a boot entry, a small configuration file, needs to be created to link the kernel, the initrd and the cmdline that will point to the correct btrds subvolume. All this management is done by sdbootutil.

Also, a new prediction needs to be done to boot without password for the new snapshot. This prediction is a policy that accounts for the different PCRs values combinations that are expected. sdbootutil will also generate those predictions using systemd-pcrlock or pcr-oracle, depending on the kind of policy that was automatically selected during enrollment.

By default it is selected the NVIndex kind of policy that systemd-pcrlock generates, but signed policy can also be generated via pcr-oracle, and this document will explain later how to migrate. The recommendation is stick with NVIndex policy, and they store the policy inside the TPM2. This kind of policy requires the use of the recovery PIN mentioned before to generate a new prediction from a system that has wrong predictions and we used the recovery password to unlock the device.

Finally, dracut-pcr-signature is a small dracut module used in initrd, that will copy the policy from the ESP into the initrd in memory, so systemd-cryptsetup can use the TPM2 policy to unseal the LUKS2 password.

Those all components works together. disk-encryption-tool will do the initial encryption of the disk and the enrollment of the different passwords and security devices the automatic decryption. sdbootutil will take care of the day by day bootloader entries management, making sure that new predictions are generated for the next boot, always measuring before the new components like the kernel and stuff. Finally with dracut-pcr-signature we guarantee that the TPM2 policy is available for the initrd before it can unlock the device, in such a way that no modification on the initrd on disk is required.

Troubleshooting

LUKS2 keyslots

List the registered keyslots in the system.

   DEV=/dev/vda3
   systemd-cryptenroll "$DEV"

This should list a "recovery" and a "tpm2" key at least. If the root password was also registered during the enrollment, then a "password" key will also be presented.

To remove an unused slot, use --wipe-slot= and to enroll a new "password" or "recovery" key use --password or --recovery-key parameters. Any alteration will require to enter one of the passwords.

PCRs

The tracked PCRs are in /etc/sysconfig/fde-tools if systemd-pcrlock is used.

The current value of those registers can be inspected with systemd-pcrlock. Because is still marked as experimental until systemd v257, it is located in /usr/lib/systemd.

The same tool can be used to inspect the event log. For each PCR extension, there is a record in the event log that register the PCR index, the extended value and some metadata that describes the purpose for the extension. This can also be inspected with the same systemd-pcrlock tool.

Policy

When pcrlock is used, a series of JSON files are generated to represent the different components that are measured during boot. They are stored in /var/lib/pcrlock.d and are orchestrated by sdbootutil.

Those component can group several measurement in one single file, representing an implicit AND operation. One component can have multiple variation, representing the OR operation.

systemd-pcrlock will try to align the current event log with the different hashes presented in the measured components, and if there is a match it will be presented in systemd-pcrlock table in one of the columns.

Note that if there is not a component associated with one entry in the current event log, we cannot make policies that includes this PCR.

The output of systemd-pcrlock when a new policy is created can be a bit confusing. For example, if a PCR prediction cannot be included in the policy (because the current component hash does not match the one in the event log, and then it cannot be mapped) the policy will be created excluding this PCR for this time.

To increase the verbosity it is advised to execute sdbootutil like this:

 SYSTEMD_LOG_LEVEL=debug sdbootutil update-policy

The generated policy (well, the policy metadata, because the real one is inside the TPM2) is living in two places. The canonical one is in /var/lib/systemd/pcrlock.json. A copy of it is done in the ESP, in /boot/efi/EFI/{systemd,opensuse}/pcrlock.json so it can be found later by dracut-pcr-signature.

If after the reboot we are requested to enter the recovery password (or a secondary one if we enrolled one extra), we can debug the reason of the mismatch.

Using systemd-pcrlock table, we can find in the "component" column rows that does not have one assigned for the PCRs that we are tracking. This table show also the expected hash, so we can find the mismatch hash checking the component registered in /var/lib/systemd/pcrlock.d.

For example, if we see that the PCR#4 (boot-loader-code) has a missing component for the last even named "efi-boot-services-application", and in the description column we see a path for the linux kernel, we known that the failing component is the kernel and the component file is "650-kernel-efi-application".

Before doing a re-enrollment (next section) we need to understand if this mismatch is a bug in the prediction code, that we forgot to update the predictions after manually changing the kernel in the ESP, or that someone altered the kernel without our permission, compromising the system!

Re-enrollment

When the prediction system fails, we need to generate a new policy for the new measurements, and replace the one stored in the TPM2. For that we need the recovery PIN, that match the recovery key (the long password generated during the enrollment to manually unlock the LUKS2 device)

When using FDE under QEMU in Tumbleweed, an update of OVMF or QEMU itself can require a re-enrollment if PCR 0 is used to seal the LUKs2 key, that is the case in the default configuration. This is because the host changed the firmware used to boot the VMs, and now the new firmware generates a different value for PCR 0 after booting.

We can force an re-enroll introducing the PIN:

 sdbootutil --ask-pin update-predictions

The environment variable PIN can also be used:

 PIN=... sdbootutil update-predictions

If for some reason we lost the recovery PIN, then we need to do the re-enrollment manually. Of course we need to have some password or FIDO2 key enrolled to access the system and unlock the device, otherwise there is nothing to do!

 # Remove the old policy from the TPM2 and from the system
 /usr/lib/systemd/systemd-pcrlock remove-policy

 # Drop the "tpm2" keyslot from the LUKS2 each device
 systemd-cryptenroll --wipe=tpm2 $DEV

 # Generate predictions and policy, and register it in the TPM2's NVRAM
 SYSTEMD_LOG_LEVEL=debug sdbootutil update-predictions

 # For each device, we update the LUKS2 header creating a new keyslot
 systemd-cryptenroll --tpm2-device=auto $DEV

Note that the re-enrollment needs to be done for all the devices.

Migration

We can migrate from NVIndex (pcrlock) to signed policy (pcr-oracle) with:

 # Remove the policy from the TPM2 and from the system
 /usr/lib/systemd/systemd-pcrlock remove-policy

 # Remove systemd-experimental package (reboot, as sdbootutil should
 # not find systemd-pcrlock)
 transactional-update pkg rm systemd-experimental

 # Remove .pcrlock files with components and variants
 rm -fr /var/lib/pcrlock.d

 # Remove pcrlock.json in /var and ESP
 rm /var/lib/systemd/pcrlock.json
 rm /boot/efi/EFI/systemd/pcrlock.json

 # Generate a RSA key pair
 pcr-oracle \
    --rsa-generate-key \
    --private-key /etc/systemd/tpm2-pcr-private-key.pem \
    --public-key /etc/systemd/tpm2-pcr-public-key.pem \
    store-public-key

 # Drop the "tpm2" keyslot from the LUKS2 device
 systemd-cryptenroll --wipe-slot=tpm2 $DEV

 # Enroll the private key
 systemd-cryptenroll \
   --tpm2-device=auto \
   --tpm2-public-key=/etc/systemd/tpm2-pcr-public-key.pem \
   --tpm2-public-key-pcrs=0,2,4,9 \
   $DEV

 # Generate new predictions with pcr-oracle
 sdbootutil update-predictions

To migrate back:

 # Install systemd-experimental (reboot, as sdbootutil should find
 # systemd-pcrlock)
 transactional-update pkg in systemd-experimental

 # Remove public and private keys fron /etc and public from ESP
 rm /etc/systemd/tpm2-pcr-private-key.pem
 rm /etc/systemd/tpm2-pcr-public-key.pem
 rm /boot/efi/EFI/systemd/tpm2-pcr-public-key.pem

 # Remove signed policy from /etc and ESP
 rm /etc/systemd/tpm2-pcr-signature.json
 rm /boot/efi/EFI/systemd/tpm2-pcr-signature.json

 # Drop the "tpm2" keyslot from the LUKS2 device
 systemd-cryptenroll --wipe-slot=tpm2 $DEV

 # Generate predictions and policy, and register it in the TPM2's NVRAM
 sdbootutil update-predictions

 # Enroll the keyslot in the LUKS2 device.  Will ask for the recovery
 # password
 systemd-cryptenroll \
   --tpm2-device=auto \
   --tpm2-pcrlock=/var/lib/systemd/pcrlock.json \
   $DEV