Matt Horan's Blog

Encrypted root disk migration for FreeBSD

I’ve had a VPS with ARP Networks for a long time now. Things were a bit different back then. The default FreeBSD installer suggested setting up multiple partitions (slices) on a disk by default. This is no longer the case. Encryption wasn’t a thing that people generally worried about. I’ve had it on my todo list for a while now to figure out how to converge to a single encrypted partition on my VPS — saving me from running out of space in /var/, and also to protect the data on the underlying “disk”. I finally worked it out a few weeks ago.

The helpful folks in the #arpnetworks IRC channel on Freenode pointed me in the right direction. A tutorial on full disk encryption got me started. However, this was only helpful on a new install. My VPS is over 10 years old, and I didn’t really want to set everything up from scratch. Also helpful was FreeBSD’s dump/restore combined with an additional disk attached to my VPS.

To start, I backed up all my existing partitions to an additional disk:

# dump -C16 -b64 -0aL -h0 -f /mnt/root.dump /
# dump -C16 -b64 -0aL -h0 -f /mnt/tmp.dump /tmp
# dump -C16 -b64 -0aL -h0 -f /mnt/var.dump /var
# dump -C16 -b64 -0aL -h0 -f /mnt/usr.dump /usr

Note that I backed up the partitions in order of volatility, figuring /usr would have the most changes by the time I finished the dump process. The -L option made it possible to dump a mounted filesystem.

Once the dump was complete, I booted from a FreeBSD install CD. When the installer popped up, I chose to start a shell.

Because I already had a had an existing partition table, I first had to destroy it before creating a new one:

# gpart destroy -F da0
# gpart create -s gpt da0

I laid out my partition table with the unencrypted boot partition first, then a swap partition, and then my root partition. To do that, I used the following commands:

# gpart add -t freebsd-boot -s 512k -a 4k da0
# gpart add -t freebsd-ufs -l bootfs -s 1g -a 1m da0
# gpart add -t freebsd-ufs -l bootfs -s 2g -a 1m da0
# gpart add -t freebsd-ufs -l encrypted -a 1m da0

I then installed the bootcode:

# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 da0

Initializing geli was easy:

# geli init -b -s 4096 da0p4

Once initialized, the device must be attached, which creates /dev/da0p4.eli:

# geli attach da0p4

Now the partitions can be formatted:

# newfs -U /dev/da0p2
# newfs -U /dev/da0p4.eli

With everything formatted, I then mounted /dev/da0p4.eli to /mnt as this is where all subsequent work would be performed:

# mount /dev/da0p4.eli /mnt

Because my backups were on /dev/da1, I had to mount them somewhere to perform the restore. I created a new directory in /mnt/ (/mnt/mnt) and mounted /dev/da1s1 there:

# mkdir /mnt/mnt
# mount /dev/da1s1 /mnt/mnt

Now I was ready to restore the filesystems atop each other. I wasn’t sure if this would work, but it did!

# cd /mnt
# restore -ruf /mnt/root.dump
# rm restoresymtable
# cd /mnt/tmp
# restore -ruf /mnt/tmp.dump
# rm restoresymtable
# cd /mnt/var
# restore -ruf /mnt/var.dump
# rm restoresymtable
# cd /mnt/usr
# restore -ruf /mnt/usr.dum
# rm restoresymtable

Note that the dump will be restore to the current directory, hence cd before each restore.

This restored /boot to my encrypted partition, which isn’t what I wanted. I had to remove the restored /boot and then set up the unencrypted partition and restore again. I discovered the -x option for restore which let me restore only the /boot directory to the unencrypted partition.

# rm -rf /mnt/boot
# mkdir /mnt/unenc
# mount /dev/ada0p2 /mnt/unenc
# cd /mnt/unenc
# restore -xuf /mnt/root.dump boot
You have not read any tapes yet.
If you are extracting just a few files, start with the last volume
and work towards the first; restore can quickly skip tapes that
have no further files to extract. Otherwise, begin with volume 1.
Specify next volume #: 1
set owner/mode for '.'? [yn] y

With everything in place, I could then add the symlink for /boot to the unencrypted partition:

# cd /mnt
# ln -s unenc/boot /mnt/boot

Having restored the backups, I then had to set up my /etc/fstab and /boot/loader.conf. I replaced /mnt/etc/fstab with the following:

# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/da0p3.eli          none            swap    sw              0       0
/dev/da0p2              /unenc          ufs     rw              1       1
/dev/da0p4.eli          /               ufs     rw              2       2

Note that while I did not geli init or geli attach /dev/da0p3, /sbin/swapon will detect the .eli suffix and encrypt the partition with a one-time key.

The FreeBSD bootloader must be configured to load geli and informed of where the encrypted root partition. I added the following to /mnt/unenc/boot/loader.conf:

geom_eli_load="YES"
vfs.root.mountfrom="ufs:ada0p3.eli"

Note that if you have a CPU with AESNI (ARP Networks VPSes do) then you should also add aesni_load="YES" to loader.conf for increased performance.

With everything in place I rebooted and was prompted to enter my password:

Enter passphrase for da0p4:

I was truly impressed at how easy this all was, especially with dump and restore, which I’d never used before! After finishing the migration, I discovered that geli init option -g which supposedly obviates the need for the unencrypted /boot partition. I’ve not tried this out myself, but perhaps that’s something to explore on a rainy day.