11 KiB
Notes for 14th of March, 2023
Today was the day I finally got to setting up both "erase your darlings"
and proper disk encryption. This general setup concept utilizes NixOS'
ability to boot off of a disk that contains only /nix and /boot, linking
appropriate devices and blocks during the boot process and deleting all state
that programs may have left over my system.
The end result, for me, was a fully encrypted that uses btrfs
snapshots to restore / to its original state on each boot.
Resources
- This discourse post
- This blog post
- This other blog post
- And this post that the previous post is based on
- Impermanence
The actual set-up (and reproduction steps)
I've had to go through a few guides before I could figure out a set up that I really like. The final decision was that I would have an encrypted disk that restores itself to its former state during boot. Is it fast? Absolutely not. But it sure as hell is cool. And stateless!
To return the root (and only the root) we use a systemd service that fires shortly after the disk is encrypted but before the root is actually mounted. That way, we can unlock the disk, restore the disk to its pristine state using the snapshot we have taken during installation and mount the root to go on with our day.
Reproduction steps
Partitioning
First you want to format your disk. If you are really comfortable with bringing parted to your pre-formatted disks, by all means feel free to skip this section. I, however, choose to format a fresh disk.
Start by partitioning the sections of our disk (sda1, sda2 and sda3) Device names might change if you're using a nvme disk, i.e nvme0p1.
# Set the disk name to make it easier
DISK=/dev/sda # replace this with the name of the device you are using
# set up the boot partition
parted "$DISK" -- mklabel gpt
parted "$DISK" -- mkpart ESP fat32 1MiB 1GiB
parted "$DISK" -- set 1 boot on
mkfs.vfat -n BOOT "$DISK"1
# set up the swap partition
parted "$DISK" -- mkpart Swap linux-swap 1GiB 9GiB
mkswap -L SWAP "$DISK"2
swapon "$DISK"2
I do in fact use swap in the civilized year of 20231. If I were a little more advanced, and if I did not disable hibernation due to overly-hardened kernel parameters, I would also be encrypting the swap to secure the hibernates... but that is currently out of my scope. You may find this desirable, however, I will not be providing instructions on that.
Encrypt your partition, and open it to make it available under /dev/mapper/enc.
cryptsetup --verify-passphrase -v luksFormat "$DISK"3 # /dev/sda3
cryptsetup open "$DISK"3 enc
Now partition the encrypted device block.
parted "$DISK" -- mkpart primary 9GiB 100%
mkfs.btrfs -L NIXOS /dev/mapper/enc
mount -t btrfs /dev/mapper/enc /mnt
# First we create the subvolumes, those may differ as per your preferences
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/nix
btrfs subvolume create /mnt/persist # some people may choose to put /persist in /mnt/nix, I am not one of those people.
btrfs subvolume create /mnt/log
Now that we have created the btrfs subvolumes, it is time for the readonly snapshot of the root subvolume.
btrfs subvolume snapshot -r /mnt/root /mnt/root-blank
# Make sure to unmount, or nixos-rebuild will try to remove /mnt and fail
umount /mnt
Mounting
After the subvolumes are created, we mount them with the options that we want.
Ideally, on NixOS, you want the noatime option 2 and zstd
compression, especially on your /nix partition.
The following is my partition layout. If you have created any other subvolumes in the step above, you will also want to mount them here. Below setup assumes that you have been following the steps as is.
# /
mount -o subvol=root,compress=zstd,noatime /dev/mapper/enc /mnt
# /home
mkdir /mnt/home
mount -o subvol=home,compress=zstd,noatime /dev/mapper/enc /mnt/home
# /nix
mkdir /mnt/nix
mount -o subvol=nix,compress=zstd,noatime /dev/mapper/enc /mnt/nix
# /persist
mkdir /mnt/persist
mount -o subvol=persist,compress=zstd,noatime /dev/mapper/enc /mnt/persist
# /var/log
mkdir -p /mnt/var/log
mount -o subvol=log,compress=zstd,noatime /dev/mapper/enc /mnt/var/log
# do not forget to mount the boot partition
mkdir /mnt/boot
mount "$DISK"1 /mnt/boot
And finally let NixOS generate the hardware configuration.
nixos-generate-config --root /mnt
The genereated configuration will be available at /mnt/etc/nixos.
Before we move on, we need to add the neededForBoot = true; to some mounted
subvolumes in hardware-configuration.nix. It will look something like this:
# Do not modify this file! It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations. Please make changes
# to /etc/nixos/configuration.nix instead.
{
config,
lib,
pkgs,
modulesPath,
...
}: {
imports = [
(modulesPath + "/installer/scan/not-detected.nix")
];
boot.initrd.availableKernelModules = ["xhci_pci" "ahci" "usb_storage" "sd_mod" "rtsx_pci_sdmmc"];
boot.initrd.kernelModules = [];
boot.kernelModules = ["kvm-intel"];
boot.extraModulePackages = [];
fileSystems."/" = {
device = "/dev/disk/by-uuid/b79d3c8b-d511-4d66-a5e0-641a75440ada";
fsType = "btrfs";
options = ["subvol=root"];
};
boot.initrd.luks.devices."enc".device = "/dev/disk/by-uuid/82144284-cf1d-4d65-9999-2e7cdc3c75d4";
fileSystems."/home" = {
device = "/dev/disk/by-uuid/b79d3c8b-d511-4d66-a5e0-641a75440ada";
fsType = "btrfs";
options = ["subvol=home"];
};
fileSystems."/nix" = {
device = "/dev/disk/by-uuid/b79d3c8b-d511-4d66-a5e0-641a75440ada";
fsType = "btrfs";
options = ["subvol=nix"];
};
fileSystems."/persist" = {
device = "/dev/disk/by-uuid/b79d3c8b-d511-4d66-a5e0-641a75440ada";
fsType = "btrfs";
options = ["subvol=persist"];
neededForBoot = true; # <- add this
};
fileSystems."/var/log" = {
device = "/dev/disk/by-uuid/b79d3c8b-d511-4d66-a5e0-641a75440ada";
fsType = "btrfs";
options = ["subvol=log"];
neededForBoot = true; # <- add this
};
fileSystems."/boot" = {
device = "/dev/disk/by-uuid/FDED-3BCF";
fsType = "vfat";
};
swapDevices = [
{device = "/dev/disk/by-uuid/0d1fc824-623b-4bb8-bf7b-63a3e657889d";}
# if you encrypt your swap, it'll also need to be configured here
];
nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
powerManagement.cpuFreqGovernor = lib.mkDefault "powersave";
}
Do keep in mind that the NixOS hardware scanner cannot pick up your mount
options. Which means that you should specifiy the options (i.e noatime) for
each btrfs volume that you have created in hardware-configuration.nix. You
can simply add them in the options = [ ] list in quotation marks. I
recommend adding at least zstd compression, and optionally noatime.
Closing Notes
And that should be all. By this point you are pretty much ready to install with your existing config. I generally use my configuration flake to boot, so there is no need to make any revisions. If you are starting from scratch, you may consider tweaking your configuration.nix before you install the system. An editor, such as Neovim, or your preferred DE/wm make good additions to your configuration.
Once it's all done, take a deep breath and nixos-install. Once the
installation is done, you'll be prompted for the root password and after that
you can reboot. Now you are running NixOS on an encrypted disk. Nice!
Next up, if you are feeling really fancy today, is to configure disk erasure and impermanence.
Impermanence
For BTRFS snapshots, I use a systemd service that goes
boot.initrd.systemd = {
enable = true; # this enabled systemd support in stage1 - required for the below setup
services.rollback = {
description = "Rollback BTRFS root subvolume to a pristine state";
wantedBy = [
"initrd.target"
];
after = [
# LUKS/TPM process
"systemd-cryptsetup@enc.service"
];
before = [
"sysroot.mount"
];
unitConfig.DefaultDependencies = "no";
serviceConfig.Type = "oneshot";
script = ''
mkdir -p /mnt
# We first mount the btrfs root to /mnt
# so we can manipulate btrfs subvolumes.
mount -o subvol=/ /dev/mapper/enc /mnt
# While we're tempted to just delete /root and create
# a new snapshot from /root-blank, /root is already
# populated at this point with a number of subvolumes,
# which makes `btrfs subvolume delete` fail.
# So, we remove them first.
#
# /root contains subvolumes:
# - /root/var/lib/portables
# - /root/var/lib/machines
btrfs subvolume list -o /mnt/root |
cut -f9 -d' ' |
while read subvolume; do
echo "deleting /$subvolume subvolume..."
btrfs subvolume delete "/mnt/$subvolume"
done &&
echo "deleting /root subvolume..." &&
btrfs subvolume delete /mnt/root
echo "restoring blank /root subvolume..."
btrfs subvolume snapshot /mnt/root-blank /mnt/root
# Once we're done rolling back to a blank snapshot,
# we can unmount /mnt and continue on the boot process.
umount /mnt
'';
};
};
You may opt in for
boot.initrd.postDeviceCommands = lib.mkBefore ''as this blog post suggests. I am not exactly sure how exactly those options actually compare, however, a systemd service means it will be accessible through the the systemd service interface, which is why I opt-in for a service.
Implications
What this implies is that certain files such as saved networks for network-manager will be deleted on each reboot. While a little clunky, Impermanence is a great solution to our problem.
Impermanence exposes to our system an environment.persistence."<dirName>" option that we can use to make certain directories or files permanent.
My module goes like this:
imports = [inputs.impermanence.nixosModules.impermanence]; # the import will be different if flakes are not enabled on your system
environment.persistence."/persist" = {
directories = [
"/etc/nixos"
"/etc/NetworkManager/system-connections"
"/etc/secureboot"
"/var/db/sudo"
];
files = [
"/etc/machine-id"
# ssh stuff
"/etc/ssh/ssh_host_ed25519_key"
"/etc/ssh/ssh_host_ed25519_key.pub"
"/etc/ssh/ssh_host_rsa_key"
"/etc/ssh/ssh_host_rsa_key.pub"
# if you use docker or LXD, also persist their directories
];
};
And that is pretty much it. If everything went well, you should now be telling your friends about your new system boasting full disk encryption and root rollbacks.
Why?
Honestly, why not?
-
I could be using
tmpfsfor/at this point in time. Unfortunately, since I share this setup on some of my low-end laptops, I've got no RAM to spare - which is exactly why I have opted out with BTRFS. It is a reliable filesystem that I am used to, and it allows for us to use a script that we'll see later on. ↩︎