Deal with FreeBSD 11.4 to 12.0 upgrade failure

I have been studying and using FreeBSD since 2004. And one of my servers saw different epochs and respective migrations along this road: csup, long nights of world & kernel build, ports src build, portupgrade, portmaster, binary freebsd-update, svnlite, pkgng, git, and a couple of hardware upgrades.

And always it was a rock solid experience — flawless upgrades got this server from 5 to 11 through the years. But, according to the following law, I’ve faced my failure day finally.

Sep, 2021. The server was on 11.4 and its EoL is by the end of Sep 2021. The server had no major updates for quite a while. Because of this fact, my ordinary human brain decided that it’s better to avoid long jumps and do the smallest step first — upgrade to 12.0. Okay, this is a personal non-critical server, why not:

And the last one started to rain with core dump messages.

My analysis told me that I was trying to go backward in time. 11.4 was released in June 2020 while 12.0 is from Dec 2018. Usually, this server had frequent updates, and there was no need to question the length of a jump. But this time the server was sitting on 11 till the very end of the branch, and things became a bit tricky.

My very first desire was to get it back to working 11.4, to do the longer jump, and upgrade it to the latest 12.2.

This host is the eldest among my servers — it has no root ZFS to cast a bectl spell to revive it. And it’s based on journaled soft-updates UFS — no snapshots respectively (hello from 2023: UFS+suj has got snapshots support!).

Okay, freebsd-update seems to have a rollback action, I had never used it before and the first attempt to run it failed. I re-checked the manual and my environment — looked like I invoked it with the correct arguments, but anyway, it could not do what I needed. Hmm…​

A non-surprise and frequent thing that happens with open source projects — we do open the sources to read them :). freebsd-update turned out to be a shell script — a sigh of relief. In addition, it is a well-written, structured, and documented shell script. I quickly got the idea of what was my roadblock. As long as the target system was in a non-usable state and I was doing this rollback running another instance of FreeBSD from a USB flash drive, my target filesystem to tinker with was mounted as /mnt, and the script’s default behavior was around usual root /. Also, I found that the script uses naming for its files imbued with some path hashing. As an outcome of my code reading, I had to simply help it with a correctly named symlink to an existing directory beneath /var/db/freebsd-update:

ln -s install.<specific-dir> "$(echo /mnt/ | sha256 -q)-rollback"

After that, it worked like a magic:

freebsd-update -b /mnt/ -d /mnt/var/db/freebsd-update/ -f /mnt/etc/freebsd-update.conf rollback

Well, another case — another +1 to experience. Also, it’s turned out to be a good idea to share my way of handling this case — others faced absolutely the same issue and my inputs were found helpful.

Copyright © Igor Ostapenko
(handmade content)