Dual boot UEFI Linux/FreeBSD

There a lot of tutorials out there explaining how to dual boot Linux and FreeBSD on legacy BIOS but not so much for UEFI only systems. So I will share my experience installing Debian 10.2 and FreeBSD 12.1 on my ThinkPad X250 in UEFI only mode. It should be easy to adapt this to other Linux distributions and other systems than FreeBSD.

This post will be soon followed by another post explaining how FreeBSD and Linux can get along with each other after being installed. But now for the installation.

First ensure in your UEFI/BIOS settings that boot is set to UEFI only and CSM disabled. You don’t want to boot the installer in legacy mode by accident.

We will use a GTP partition table and the create the following partitions:

  1. EFI System Partition (ESP): To store the UEFI bootloaders
  2. SWAP + ext4: For Linux
  3. SWAP + UFS: For FreeBSD

Note that we could technically use the same SWAP for both FreeBSD and Linux. Still I prefer to use 2 SWAP partitions and use both of them in each OS. It’s a matter of preference I guess.

Install Linux and FreeBSD

Install Linux

We start installing Debian so that Linux stays in charge of GRUB. We do so because it’s frequent on Linux to have multiple kernel variants, for instance a more recent version of the kernel, custom or a RT patched kernel. So we let Linux’s package manager handle all of that.

At the partitioning step, select “Manual partitioning” and create a GPT partition table. For UEFI to function properly, we need a EFI System Partition (ESP), 500MB should be more than enough for this. Then a Linux swap partition and a Linux ext4 partition for the data. Leave some space unallocated for FreeBSD.

It is possible to use Linux’s swap partition in FreeBSD. More about that later. For now we will let each OS have its own swap partition.

Complete the Debian installation. It should install grub bootloader in the EFI partition. Check that Debian boots properly. Then start the FreeBSD install.

Install FreeBSD

There used to be a separate installation image for FreeBSD UEFI. This is not the case anymore, so you can use the AMD64 memstick image for 12.1-RELEASE on the FreeBSD download page.

Boot the installer and go ahead up to the partitioning step. Use the space you left unallocated for the freebsd-swap and freebsd-ufs partitions. The installer will complain that an EFI partition is required for the system to work properly and propose to create it. Ignore this as the partition was already created under Linux. It is weird though that the FreeBSD installer does not detect this, but there is a FreeBSD forum post about this issue.

Proceed and complete the FreeBSD installation. Then reboot into Debian to configure the dual boot.

GRUB dual boot

It is now time to tell Linux’s GRUB about our newly installed FreeBSD system.

We created the swap partitions before the data partitions for each OS, so to resume our partition table we now have:

  1. EFI System Partition
  2. Linux SWAP
  3. Linux ext4
  4. FreeBSD SWAP
  5. FreeBSD UFS

So our FreeBSD partition is (hd0, gpt5) in GRUB parlance. You may need to adapt this to your own partition scheme though. Once in Debian as root edit /etc/grub.d/40_custom and after the comment add:


menuentry 'FreeBSD' {
insmod ufs2
set root='(hd0,gpt5)'
chainloader /boot/loader.efi
}

Then update grub with update-grub2, finally reboot and select FreeBSD in the GRUB menu. You can now boot both Linux and FreeBSD.

FreeBSD aware UEFI

It is now possible to boot both Debian and FreeBSD from GRUB. However it is not yet possible to boot FreeBSD directly from UEFI. To do so we need to copy the FreeBSD UEFI loader in to EFI partition and register it. Debian already mounts the EFI partition but FreeBSD doesn’t, so for the fun of it, let’s manage all that under FreeBSD and install the FreeBSD UEFI loader. /dev/ada0p1 is the EFI partition, but you may need to adapt this to your partition scheme though.

# We mount the EFI partition on /boot/efi similarly to Linux.
mkdir /boot/efi
echo '/dev/ada0p1 /boot/efi msdosfs rw,noatime 0 0' >> /etc/fstab
mount /boot/efi

# Install the FreeBSD UEFI loader.
mkdir /boot/efi/EFI/freebsd
cp /boot/boot1.efi /boot/efi/EFI/freebsd/bootx64.efi

Now let’s create an UEFI entry for this loader. Note that this is for FreeBSD’s efibootmgr, not the Linux’s one.

# Create the boot variable.
efibootmgr -c -l /boot/efi/EFI/freebsd/bootx64.efi -L "FreeBSD"

# Check the variable number for the new boot variable and activate it.
efibootmgr
efibootmgr -a 15

# Change the boot order to leave Debian and GRUB in charge.
efibootmgr -o 14,15

Time to reboot! Select the boot menu with (generally with F12, at least on my ThinkPad X250) and FreeBSD should appear. Select it and it should boot FreeBSD directly.

You are done! Next time how to let FreeBSD and Linux talk to each other.

Intel kernel panic

If you recently had your FreeBSD 12.1 running ThinkPad X250 (or any other laptop with a Intel GPU) crashing systematically on boot, it may be because of the i915kms module from the graphics/drm-kmod port. If you did install this or drm-fbsd12.0-kmod through the packages, you should know that the binary package is only compatible with FreeBSD 12.0. If you want this to work (that is, not to cause a kernel panic), you should compile the port manually [1]. In other words:

portsnap fetch update
cd /usr/ports/graphics/drm-fbsd12.0-kmod
make install clean

Bhyve OpenBSD on FreeNAS

Lately I’ve been playing with bhyve on FreeBSD and FreeNAS in the prospect of spinning up small VMs for a mini compile farm (and just as an excuse to play around). In this post I will share my experience in installing OpenBSD 6.4 as a bhyve UEFI VM on FreeNAS-11.2 through the new GUI.

This post is divided in three parts. First the creation of the VM through FreeNAS new GUI. Then the installation of OpenBSD from boot to finish. Lastly some bits of documentation and related posts that were helpful along the way.

Create the VM

The installation will go through serial instead of VNC. Also, instead of the CDROM ISO we use a disk image for the installation.

In the FreeNAS GUI, selects Virtual Machines > ADD, then proceed with the wizard. Some parameters of the wizard are pretty obvious, so there won’t be an explanation for each one of them. If need be, check the FreeNAS doc on creating VMs.

Guest Operating System, not exactly sure what this does. Guess it’s basically a template for the next steps of the wizard. Since this is a BSD system, select FreeBSD as this is the closest there is.

We don’t need VNC, so make sure that Enable VNC is unchecked. As for Boot Method, you can select UEFI, you don’t need UEFI-CSM.

For the hard disk in Select Disk Type and similarly for the network interface in Adapter Type, select a VirtIO type.

For the Installation Media, leave it blank. This is for an ISO image but OpenBSD’s installXX.iso didn’t work so we are going to use a raw disk image instead.

Go ahead and create the VM. But don’t start it yet. We have to add a raw disk device.

Fetch installXX.fs from OpenBSD download page, and put it somewhere on the FreeNAS host. For the following, I’ll assume this is install64.fs.

Then select Devices > ADD for the VM. Select Type: Raw File and Raw File: the location of install64.fs on the FreeNAS host. Also Mode: VirtIO; Device Order: 1003 (the device order is important, otherwise the OpenBSD install would incorrectly guess the target device); Raw filesize: 1 (which means 1GB).

Now you can start the VM and open the Serial console. You should be greeted by:

>> OpenBSD/amd64 BOOTX64 3.40
boot>

Time to install OpenBSD!

Install OpenBSD

You are on the serial console with the OpenBSD install bootloader waiting for you. On the console boot>, type set tty com0, then boot.

Do not directly start the installation, we first have to create the EFI partition, so select (S)hell instead. At the command prompt, create the EFI partition as described below, then start the install.

## Check that sd0 is the correct target device
# disklabel -p M sd0
...
16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:         15625.0M                0  unused

## Initialise a GPT partition table with the special boot partition on sd0.
# fdisk -iyg -b 960 sd0
Writing MBR at offset 0.
Writing GPT.

## Start the installer
# install

At the partitioning step, select (O)penBSD area. We will setup a single root partition layout. You should know that the OpenBSD bootloader likes its root partition on slice a of the first hard disk, so we create the layout that way.

# Check the current partition
> p M
OpenBSD area: 1024-31999937; size: 15624.5M; free: 15624.5M
#                size           offset  fstype [fsize bsize   cpg]
  c:         15625.0M                0  unused                    
  i:             0.5M               64   MSDOS 

# Start with the root partition.
# Again, the bootloader likes it that way.
> a
...
size: {your-root-partition-size}M
FS type: 4.2BSD
mount point: /

# Now the swap partition
> a
...
FS type: swap

# Check again
> p M
OpenBSD area: 1024-31999937; size: 15624.5M; free: 0.0M
#                size           offset  fstype [fsize bsize   cpg]
  a:         15493.9M             1024  4.2BSD   2048 16384     1 # /
  b:           130.5M         31732576    swap               
  c:         15625.0M                0  unused                 
  i:             0.5M               64   MSDOS

# Quit and save
> q
Write new label?: y

Proceed with the installation and the file sets and once you are done, reboot in your new system. Remove the Raw File device from the VM and on the VM itself, use syspatch to patch the base system. Finally reboot and you are done!

Relevant bits of documentation

Drop TCP connections

On FreeBSD you can drop existing TCP connection using the tcpdrop command. For instance you can drop all ESTABLISHED connections using tcpdrop -s ESTABLISHED. Or you can even list them all with:

$ tcpdrop -la
tcpdrop ::1 59298 ::1 1180
tcpdrop 10.0.0.10 59299 163.172.87.245 80
tcpdrop 10.0.0.10 59300 163.172.87.245 22
tcpdrop 10.0.0.10 59301 96.47.72.84 443

Notice the fun thing here. Those are actual commands that you can use to drop the connections. In fact you can use this to filter which connection you want to drop. For example:

# Drop all but SSH connections
tcpdrop -la | grep -vw 22 | sh

# Drop all incoming HTTP connections
tcpdrop -la | grep -v " 80 " | sh

# Drop all connections to a specific IP
tcpdrop -la | grep -vw 8.8.8.8 | sh

This can be useful for instance on a desktop when you just switched interface, or say just started a VPN daemon, and want all prior TCP connections not originating from your new addresses to be killed. Then you would just add those IP you would like to keep and filter them out:

# List of IPs you want to keep
echo 192.168.1.122 > keep-ip.txt
echo 10.0.0.10 >> keep-ip.txt

tcpdrop -la | grep -Ev "(::1|127.0.0.1)" | grep -vwf keep-ip.txt | sh

Automount not working with FreeBSD 12

FreeBSD 12 is out. This is great! However I had the surprise to find that the automount feature didn’t work in KDE, probably also Gnome, XFCE and any other desktop environment that provide such a feature.

The culprit was easy to find, the Hardware Abstraction Layer has not yet updated to the peculiarities of the latest FreeBSD release.

See, when HAL tries to mount a vfat filesystem on FreeBSD, it adds by default the large option which according to FreeBSD 11.2 mount_msdosfs’s manpage provide support for very large files (>128GB). This option, however, was removed in FreeBSD 12. Thus automount fails.

To temporarily fix this, edit /usr/local/share/hal/fdi/policy/10osvendor/20-storage-methods.fdi. Then remove the large option in the vfat match for FreeBSD. That is:

  <match key="volume.fstype" string="vfat">
    <match key="/org/freedesktop/Hal/devices/computer:system.kernel.name" string="Linux">
      ...
  </match>
  <match key="/org/freedesktop/Hal/devices/computer:system.kernel.name" string="FreeBSD">
    ...
    <!-- <append key="volume.mount.valid_options" type="strlist">large</append> -->
  </match>

This was already reported in #221709.

Don’t forget the pipe subshell

This is a common error while using pipe over while loops. Consider this shell snippet:

#!/bin/sh

cat file.txt | while read line
do
  echo "inside loop"
  exit 1
done

echo "outside loop"
exit 0

You’d expect the script to exit on the first line in file.txt. However execute this script and you have:

inside loop
outside loop

It is as if the exit 1 inside the loop is ignored. Another example:

#!/bin/sh

a=0
cat file.txt | while read line
do
  echo "inside loop"
  a=1
done

echo "outside loop"
echo "a=$a"

Here you’d expect the value of a to be 1 at the end of the script. Instead, if you execute this you have:

inside loop
outside loop
a=0

It’s as if the variable a isn’t even updated. In fact it is, though only inside the loop. So what is happening here?

The pipe (|) you use to feed the loop creates a subshell. In fact this is really just another process. So the exit 1 or a=1 only apply to these piped processes.

How can you fix that?
In the simple case presented above, you can simply use file redirection:

while read line
do
  ...
done < file.txt

But what if you really want to feed the loop with the output of another process. Like you would do with find for instance.

If you use bash you can use process substitution as described here. But you shouldn’t use bash for scripting anyway. For shell scripting you might be tempted to use a temporary file to store the process output:

# Use a temporary file.
tmp=$(mktemp)
find . > $tmp
while read line
do
  ...
done < $tmp 
rm $tmp

However this consumes disk space, and the loop only starts after the find process exited. Another option would be to use a named fifo:

fifo=$(mktemp -u)
mkfifo $fifo
find . > $fifo &

while read file
do
  ...
done < $fifo
rm $fifo

This time you create a single file, yet no disk space is used (apart for the fifo inode itself). Also the find command is a child process, so the loop reads find output as it comes.

Although the version above already works as it should, you may want to use an anonymous fifo. This way you only need to create a fifo file, although you can delete it immediatly. You can achieve this with a little help from our beloved file descriptor 3.

fifo=$(mktemp -u)

# Create fifo
mkfifo $fifo

# Create fd 3 and unlink fifo file.
exec 3<> $fifo
rm $fifo

# Redirect find to fd 3.
find . >&3 &

# Feed fd 3 to while loop.
while read line
do
  ...
done <&3 # Close fd 3. exec 3>&-

Epson 3490 Scanner

This is an information that tends to be forgotten on the Internet, so I’m publishing it here. How to get an Epson Perfection 3490 Photo scanner running under Linux, FreeBSD or whatever. Paths may change on your system, so you may need to adapt the instructions below.

1. Install xsane.

2. Download the Epson firmwares. For the Epson 3490, you need esfw52.bin. You may find this file on Internet, although it tends to disappear. But in any case you can also find this file here.

3. Uncompress the firmwares. That is, sudo tar -Jxvf epson-firmwares.tar.xz -C /usr/local/share/sane.

4. Modify /usr/local/etc/sane.d/snapscan.conf, change the firmware line to point to the esfw52.bin firmware. That is following the commands above, change the firmware line to firmware /usr/local/share/sane/epson-firmwares/esfw52.bin.

If you are running FreeBSD

You should still ensure that you can use the scanner as a normal user.

5.Let’s change the owner of the scanner so that it’s available to users in the saned group. Create /etc/devd/saned.conf and add:

notify 100 {
  match "system" "USB";
  match "subsystem" "INTERFACE";
  match "type" "ATTACH";
  match "cdev" "ugen[0-9].[0-9]";
  match "vendor" "0x04b8";
  match "product" "0x0122";
  action "chown -L cups:saned /dev/$cdev && chmod -L 660 /dev/$cdev";
};

Notice the 0x4b8:0x0122, identifying the scanner USB device which you can get from the lsusb command while the scanner is plugged in.

6.Restart devd with service devd restart.

7.Add yourself to the saned group with sudo pw groupmod saned -m

8.You may need to log in again so that new group changes are taken into account.