Beyond dual boot, make Linux and FreeBSD friends

This is a continuation of a previous post on dual boot UEFI Linux/FreeBSD.

Dual boot is not enough and we want both FreeBSD and Linux to get along for the better. To this end we will discuss several aspect of making both OS recognize each other and work in parallel.

Note that this post in particular is subject to changes. These things evolve rapidly so some details may not be accurate, not up-to-date or differ from your specific setup. So please feel free to comment/update/add ideas/point out errors or missing details. Thanks!

We will look at the following aspect:

Access each other data

The first step is to recall your partition layout and what it looks like on FreeBSD and Linux. Take time to note it on a piece of paper. That’s especially useful if you have extra hard disks with additional data partitions. To recall in the previous post we had:

Name FreeBSD Linux
EFI /dev/ada0p1 /dev/sda1
Linux SWAP /dev/ada0p2 /dev/sda2
Linux ‘/’ (ext4) /dev/ada0p3 /dev/sda3
FreeBSD SWAP /dev/ada0p4 /dev/sda4
FreeBSD ‘/’ (UFS) /dev/ada0p5 /dev/sda5

FreeBSD and Linux can both access their counterpart ext4 and UFS partition. In particular, FreeBSD can also write to Linux’s ext4 partitions. Linux supports writing to UFS in theory, but I would strongly recommend against that. Last time I tested, it only completely wrecked the UFS partition and had to reformat it. In fact, I’d recommend that you mount the counterpart OS partition in read-only to avoid messing with anything. Eventually setup another shared ext4 partition that you access in writing from both Linux and FreeBSD. At least if something goes wrong, you only loose that.

As for the swap partitions, both OS can use each other partition. Linux however requires a special signature which can be created with mkswap. However I don’t think this is still required and it works fine without it.

We will add the mount point in /mnt, that’s as good a place as any. Of course everything below must be done as root.


Create the mount point for Linux:

mkdir /mnt/linux

We want to mount the second (Linux swap) and third (Linux ext4) partitions, that is for FreeBSD /dev/ada0p2 and /dev/ada0p3. Add the partitions to the end of /etc/fstab:

/dev/ada0p2 none swap sw 0 0
/dev/ada0p3 /mnt/linux ext2fs failok,ro 0 0

Notice the failok option for /mnt/linux. It means that the FreeBSD boot should not fail if the ext4 partition fails to mount. Otherwise, if the partition was marked as dirty by Linux (for instance you did a hard reboot) and then you reboot directly into FreeBSD, mounting the ext4 partition would fail because it requires an fsck.


Create the mount point for FreeBSD:

mkdir /mnt/freebsd

We want to mount the fourth (FreeBSD swap) and fifth (FreeBSD UFS) partitions, that is for Linux /dev/sda4 and /dev/sda5. Add the partitions to the end of /etc/fstab:

/dev/sda4 none swap sw 0 0
/dev/sda5 /mnt/freebsd ufs nofail,ro,ufstype=ufs2 0 0

Notice the nofail similar to failok in FreeBSD.

Share documents

Now that both OS can access each other data, it’s time to see if we can put this to some use. A first thing that you might do is to share some parts of your home directory. Of course this will be read-only.

What I personally do is selecting the OS that I use most frequently. This one contains everything. Then on the other OS, I use symlinks to some part of my home directory. For instance on Linux:

  • /home/user/Music    -> /mnt/freebsd/home/user/Music
  • /home/user/Pictures -> /mnt/freebsd/home/user/Pictures
  • /home/user/Videos   -> /mnt/freebsd/home/user/Videos

Those don’t change that often when I’m on Linux, but I least I got to listen to music, watch movies and can access pictures.

Share ssh keys

If you frequently use ssh to access your laptop or rsync+ssh to sync your documents, you will soon find yourself with ssh complaining that the host key has changed on the same host. Of course FreeBSD and Linux will both have a separate set of ssh host keys. Thankfully we can use the same key on both.

Suppose that we use FreeBSD’s ssh host keys. On Linux, go into /etc/ssh:

# Remove Linux's ssh host keys
rm ssh_host_*key*

# Link FreeBSD's ssh host keys
for key in /mnt/freebsd/etc/ssh/ssh_host_*key*
  ln -s "$key"

Don’t forget to service ssh restart. Now you can access both Linux and FreeBSD with ssh as if it was the same host (which it is actually).

Share WPA supplicant (WiFi)

Same principle now for WPA supplicant configuration file. However it’s not as simple as it was for ssh. You see, the WPA supplicant configuration needs different options for the control socket on FreeBSD and Linux. Unfortunately, wpa_supplicant.conf does not allow for file include. Actually it should be possible to arrange FreeBSD and Linux so that the same wpa_supplicant.conf is used on both OS. But this option allows for more flexibility. So here is what I do.

First I create /etc/wpa in both FreeBSD and Linux. Then I edit /etc/wpa/local.conf with the wpa_supplicant options specific to this OS. Then I use this small script to select a particular profile and create the appropriate wpa_supplicant.conf.


if [ ! -r "$1" ]
  echo "error: cannot read '$1'"
  exit 1

cat /etc/wpa/local.conf > /etc/wpa_supplicant.conf
cat "$1" >> /etc/wpa_supplicant.conf

The idea behind those profiles is to restrict scanning of new networks depending on the situation. For instance you can have one profile for your home, one for your working place, one when you go abroad. It’s easier to organize your configuration that way and also avoids to send probe requests on the air that can disclose information about you.

Thus on FreeBSD, I create /etc/wpa/profiles along with the various profiles and on Linux, I just link to it.

Share nullmailer

If you happen to use nullmailer as your local MTA, you can share your smtp credentials too. But there is a catch. The remotes file in the nullmail configuration must be owned by the nullmailer user/group. This user/group is different on Linux than it is on FreeBSD (nullmail on FreeBSD, mail on Linux). Fortunately when you mount a filesystem, it only cares about the UID/GID, not the actual user/group name. So if we change the UID/GID of the user/group mail to match the UID/GID of the nullmail user/group on FreeBSD, it will appear as the same user but with a different name on each OS. That’s what we’ll do on Linux, we will change the UID/GID of mail to match the one on FreeBSD.

First, let’s list all files owned by user mail:

find / -xdev -user mail > user
find / -xdev -group mail > group

Second, check that we won’t mess around too much by changing these files:

$ cat user group | sort | uniq

Seems OK. Now let’s find out the UID/GID of the nullmail user group on FreeBSD:

$ cat /mnt/freebsd/etc/passwd | grep nullmail
nullmail:*:522:522:Nullmailer Mail System:/var/spool/nullmailer:/bin/sh
$ cat /mnt/freebsd/etc/group | grep nullmail

The UID/GID is 522:522 on FreeBSD. We will change user/group mail on Linux to UID/GID 522:

# Stop nullmailer so that we can change the UID/GID
service nullmailer stop

# Change UID/GID
usermod -u 522 mail
groupmod -u 522 mail

# The files are still owned by the old UID/GID.
# We change that
cat user | while read file
  chown mail: "$file"
cat group | while read file
  chown :mail "$file"

# Clean
rm user group

Now we link nullmailer configuration into Linux:

rm -rf /etc/nullmailer
ln -s /mnt/freebsd/usr/local/etc/nullmailer /etc/nullmailer

You should be able to service nullmailer restart now.

Dual boot UEFI Linux/FreeBSD

There a lot of tutorials out there explaining how to dual boot Linux and FreeBSD on legacy BIOS but not so much for UEFI only systems. So I will share my experience installing Debian 10.2 and FreeBSD 12.1 on my ThinkPad X250 in UEFI only mode. It should be easy to adapt this to other Linux distributions and other systems than FreeBSD.

This post will be soon followed by another post explaining how FreeBSD and Linux can get along with each other after being installed. But now for the installation.

First ensure in your UEFI/BIOS settings that boot is set to UEFI only and CSM disabled. You don’t want to boot the installer in legacy mode by accident.

We will use a GTP partition table and the create the following partitions:

  1. EFI System Partition (ESP): To store the UEFI bootloaders
  2. SWAP + ext4: For Linux
  3. SWAP + UFS: For FreeBSD

Note that we could technically use the same SWAP for both FreeBSD and Linux. Still I prefer to use 2 SWAP partitions and use both of them in each OS. It’s a matter of preference I guess.

Install Linux and FreeBSD

Install Linux

We start installing Debian so that Linux stays in charge of GRUB. We do so because it’s frequent on Linux to have multiple kernel variants, for instance a more recent version of the kernel, custom or a RT patched kernel. So we let Linux’s package manager handle all of that.

At the partitioning step, select “Manual partitioning” and create a GPT partition table. For UEFI to function properly, we need a EFI System Partition (ESP), 500MB should be more than enough for this. Then a Linux swap partition and a Linux ext4 partition for the data. Leave some space unallocated for FreeBSD.

It is possible to use Linux’s swap partition in FreeBSD. More about that later. For now we will let each OS have its own swap partition.

Complete the Debian installation. It should install grub bootloader in the EFI partition. Check that Debian boots properly. Then start the FreeBSD install.

Install FreeBSD

There used to be a separate installation image for FreeBSD UEFI. This is not the case anymore, so you can use the AMD64 memstick image for 12.1-RELEASE on the FreeBSD download page.

Boot the installer and go ahead up to the partitioning step. Use the space you left unallocated for the freebsd-swap and freebsd-ufs partitions. The installer will complain that an EFI partition is required for the system to work properly and propose to create it. Ignore this as the partition was already created under Linux. It is weird though that the FreeBSD installer does not detect this, but there is a FreeBSD forum post about this issue.

Proceed and complete the FreeBSD installation. Then reboot into Debian to configure the dual boot.

GRUB dual boot

It is now time to tell Linux’s GRUB about our newly installed FreeBSD system.

We created the swap partitions before the data partitions for each OS, so to resume our partition table we now have:

  1. EFI System Partition
  2. Linux SWAP
  3. Linux ext4
  4. FreeBSD SWAP
  5. FreeBSD UFS

So our FreeBSD partition is (hd0, gpt5) in GRUB parlance. You may need to adapt this to your own partition scheme though. Once in Debian as root edit /etc/grub.d/40_custom and after the comment add:

menuentry 'FreeBSD' {
insmod ufs2
set root='(hd0,gpt5)'
chainloader /boot/loader.efi

If you want, you can also configure FreeBSD as the default entry by editing /etc/default/grub and change DEFAULT=0 to DEFAULT=FreeBSD.

Then update grub with update-grub2, finally reboot and select FreeBSD in the GRUB menu. You can now boot both Linux and FreeBSD.

FreeBSD aware UEFI

It is now possible to boot both Debian and FreeBSD from GRUB. However it is not yet possible to boot FreeBSD directly from UEFI. To do so we need to copy the FreeBSD UEFI loader in to EFI partition and register it. Debian already mounts the EFI partition but FreeBSD doesn’t, so for the fun of it, let’s manage all that under FreeBSD and install the FreeBSD UEFI loader. /dev/ada0p1 is the EFI partition, but you may need to adapt this to your partition scheme though.

# We mount the EFI partition on /boot/efi similarly to Linux.
mkdir /boot/efi
echo '/dev/ada0p1 /boot/efi msdosfs rw,noatime 0 0' >> /etc/fstab
mount /boot/efi

# Install the FreeBSD UEFI loader.
mkdir /boot/efi/EFI/freebsd
cp /boot/boot1.efi /boot/efi/EFI/freebsd/bootx64.efi

Now let’s create an UEFI entry for this loader. Note that this is for FreeBSD’s efibootmgr, not the Linux’s one.

# Create the boot variable.
efibootmgr -c -l /boot/efi/EFI/freebsd/bootx64.efi -L "FreeBSD"

# Check the variable number for the new boot variable and activate it.
efibootmgr -a 15

# Change the boot order to leave Debian and GRUB in charge.
efibootmgr -o 14,15

Time to reboot! Select the boot menu with (generally with F12, at least on my ThinkPad X250) and FreeBSD should appear. Select it and it should boot FreeBSD directly.

You are done! Next time how to let FreeBSD and Linux talk to each other.

Intel kernel panic

If you recently had your FreeBSD 12.1 running ThinkPad X250 (or any other laptop with a Intel GPU) crashing systematically on boot, it may be because of the i915kms module from the graphics/drm-kmod port. If you did install this or drm-fbsd12.0-kmod through the packages, you should know that the binary package is only compatible with FreeBSD 12.0. If you want this to work (that is, not to cause a kernel panic), you should compile the port manually [1]. In other words:

portsnap fetch update
cd /usr/ports/graphics/drm-fbsd12.0-kmod
make install clean

Bhyve OpenBSD on FreeNAS

Lately I’ve been playing with bhyve on FreeBSD and FreeNAS in the prospect of spinning up small VMs for a mini compile farm (and just as an excuse to play around). In this post I will share my experience in installing OpenBSD 6.4 as a bhyve UEFI VM on FreeNAS-11.2 through the new GUI.

This post is divided in three parts. First the creation of the VM through FreeNAS new GUI. Then the installation of OpenBSD from boot to finish. Lastly some bits of documentation and related posts that were helpful along the way.

Create the VM

The installation will go through serial instead of VNC. Also, instead of the CDROM ISO we use a disk image for the installation.

In the FreeNAS GUI, selects Virtual Machines > ADD, then proceed with the wizard. Some parameters of the wizard are pretty obvious, so there won’t be an explanation for each one of them. If need be, check the FreeNAS doc on creating VMs.

Guest Operating System, not exactly sure what this does. Guess it’s basically a template for the next steps of the wizard. Since this is a BSD system, select FreeBSD as this is the closest there is.

We don’t need VNC, so make sure that Enable VNC is unchecked. As for Boot Method, you can select UEFI, you don’t need UEFI-CSM.

For the hard disk in Select Disk Type and similarly for the network interface in Adapter Type, select a VirtIO type.

For the Installation Media, leave it blank. This is for an ISO image but OpenBSD’s installXX.iso didn’t work so we are going to use a raw disk image instead.

Go ahead and create the VM. But don’t start it yet. We have to add a raw disk device.

Fetch installXX.fs from OpenBSD download page, and put it somewhere on the FreeNAS host. For the following, I’ll assume this is install64.fs.

Then select Devices > ADD for the VM. Select Type: Raw File and Raw File: the location of install64.fs on the FreeNAS host. Also Mode: VirtIO; Device Order: 1003 (the device order is important, otherwise the OpenBSD install would incorrectly guess the target device); Raw filesize: 1 (which means 1GB).

Now you can start the VM and open the Serial console. You should be greeted by:

>> OpenBSD/amd64 BOOTX64 3.40

Time to install OpenBSD!

Install OpenBSD

You are on the serial console with the OpenBSD install bootloader waiting for you. On the console boot>, type set tty com0, then boot.

Do not directly start the installation, we first have to create the EFI partition, so select (S)hell instead. At the command prompt, create the EFI partition as described below, then start the install.

## Check that sd0 is the correct target device
# disklabel -p M sd0
16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:         15625.0M                0  unused

## Initialise a GPT partition table with the special boot partition on sd0.
# fdisk -iyg -b 960 sd0
Writing MBR at offset 0.
Writing GPT.

## Start the installer
# install

At the partitioning step, select (O)penBSD area. We will setup a single root partition layout. You should know that the OpenBSD bootloader likes its root partition on slice a of the first hard disk, so we create the layout that way.

# Check the current partition
> p M
OpenBSD area: 1024-31999937; size: 15624.5M; free: 15624.5M
#                size           offset  fstype [fsize bsize   cpg]
  c:         15625.0M                0  unused                    
  i:             0.5M               64   MSDOS 

# Start with the root partition.
# Again, the bootloader likes it that way.
> a
size: {your-root-partition-size}M
FS type: 4.2BSD
mount point: /

# Now the swap partition
> a
FS type: swap

# Check again
> p M
OpenBSD area: 1024-31999937; size: 15624.5M; free: 0.0M
#                size           offset  fstype [fsize bsize   cpg]
  a:         15493.9M             1024  4.2BSD   2048 16384     1 # /
  b:           130.5M         31732576    swap               
  c:         15625.0M                0  unused                 
  i:             0.5M               64   MSDOS

# Quit and save
> q
Write new label?: y

Proceed with the installation and the file sets and once you are done, reboot in your new system. Remove the Raw File device from the VM and on the VM itself, use syspatch to patch the base system. Finally reboot and you are done!

Relevant bits of documentation

Drop TCP connections

On FreeBSD you can drop existing TCP connection using the tcpdrop command. For instance you can drop all ESTABLISHED connections using tcpdrop -s ESTABLISHED. Or you can even list them all with:

$ tcpdrop -la
tcpdrop ::1 59298 ::1 1180
tcpdrop 59299 80
tcpdrop 59300 22
tcpdrop 59301 443

Notice the fun thing here. Those are actual commands that you can use to drop the connections. In fact you can use this to filter which connection you want to drop. For example:

# Drop all but SSH connections
tcpdrop -la | grep -vw 22 | sh

# Drop all incoming HTTP connections
tcpdrop -la | grep -v " 80 " | sh

# Drop all connections to a specific IP
tcpdrop -la | grep -vw | sh

This can be useful for instance on a desktop when you just switched interface, or say just started a VPN daemon, and want all prior TCP connections not originating from your new addresses to be killed. Then you would just add those IP you would like to keep and filter them out:

# List of IPs you want to keep
echo > keep-ip.txt
echo >> keep-ip.txt

tcpdrop -la | grep -Ev "(::1|" | grep -vwf keep-ip.txt | sh

Today’s movie: Alita: Battle Angel

Alita: Battle Angel by Robert Rodriguez (one of the co-director of Sin City) along with James Cameron as co-screenplay. This is based on Gunnm, a classic gory cyberpunk manga (or so I’ve been told).

Even if you didn’t know about Gunnm before, you could tell that this was overall faithful to the original material (this was in fact confirmed by someone that did know the manga later on). Clearly it has been smoothed down for a wider audience, but it allows itself to be continuously very suggestive and that’s cool. As long as you are for the least imaginative, you won’t be deceived.

It goes fast, and sure you can feel that. Clearly you know at the end that there is a lot that ought to happen in between. One thing that you can blame on the movie, however, is that it tries a lot to stick to a classical narrative of the hero journey to self-discovery. Now, fitting a large and complex story in a 2h30 long movie is hard. Codifying this so it appeals to the largest number of people is probably a lot of a heck harder.

The movie also features some good actors. Among them is Christoph Waltz, this guy is so versatile, you can never tell if he will play the ultimate bad-ass villain or the absolute good-guy or anything else in between. And in this case, such a versatility fits well. But that’s not only him, all the cast fits and contributes to a coherent whole. What could you ask more from a blockbuster movie?

All in all, was going there not knowing what to expect, and now I find myself in the mood of reading this manga as soon as possible. Thanks I guess!

Today’s movie: The horror movie night

Yester-Yesterday was the 7th of January which is probably the best date for an Halloween horror movie night. So we went forth and selected four movies.

In the mouth of madness by John Carpenter with the underrated Sam Neill. A recursive horror fable that was most enjoyable and at times funny. It’s crazy the amount of smoke that got lighten up in a Carpenter movie. But that may be one of the reason behind some smoking. It cuts time into separate definite moments. It offers a pace when everything else would be tirelessly continuous.

Beyond the black rainbow by Panos Cosmatos. What the hell was this? Incredibly aesthetic, still it is not the kind of movie you understand on your first viewing but instead infuses slowly as you dive yourself into it. It’s like Kubrick’s 2001 and Lynch’s Eraserhead somehow decided to make out in the 80s after taking way too much drugs. The end was maybe a little disappointing. But I will no doubt enjoy watching it again.

Audition (オーディション) by Takashi Miike. One wonders for a moment if this is really an horror movie. But when it gets into it, you won’t be disappointed! If I had to resume in four words, I would say

Kili, kili, kili, kili… :3

Viewers be warned, however, this movie may contain some traces of dinosaurs.

The Witch by Robert Eggers. This movie takes us in 1630 to follow a family of settlers banished and isolated on the edge of the woods. It follows how in harsh conditions such a close community can rot from within and destroy itself. In a sense, it is similar to The Village but takes the story on a completely different level. The particularly heavy atmosphere increases with a constant pace and the religious aspect underlying the life of the family is undeniably really creepy.

One thing that I particularly liked was the attention paid to the realism of life at that time. For instance, the use of Old English gave it an authentic charm. But if it wasn’t for the subtitles we wouldn’t have understood a darn thing.

Why is it when thou dost a wrong, I be a-washing Fathers clothes like a slave, and thou art playing idle? It is, thou thing! How I crave to sink my teeth into thy pink flesh. If ever thou tellst thy mother of this, I will witch thee and thy mother!

I want to speak like this all day long now.