Replace USB boot key on FreeNAS

You don’t really install FreeNAS as you do with some other OS. Instead you have a device (often an USB key) that boots FreeNAS which itself manages your data storage. Indeed why wasting some precious SATA port for a boot device when all you need is a little USB key? Especially when thits boot device is less than 2GB in size.

The problem with USB keys is that they wear out over time. So it inevitably comes a time when it must be replaced. This is what happened to me a few days ago.

The freenas-boot pool switched to a DEGRADED state because checksum error happened too frequently. Fortunately a zpool scrub freenas-boot detected no error, and it seems that zfs was still able to correct those. So instead of reflashing a new USB key and restoring my config backup, I could from FreeNAS itself create a new bootable USB key and replace replace the faulty device in the freenas-boot pool.

Let’s login on FreeNAS and plug in your new USB key. We first find and verify what is the device name for the USB key, in my case it’s da2:

$ gpart part show da2
=>       1  60088319  da2  MBR  (29G)
         1        31       - free -  (16K)
        32  60088288    1  fat32lba  (29G)

You see that MBR table and a FAT32 partition, in my case it can only be the new USB key. We will replace that with GPT partitions. This will contain a boot partition (MBR boot or EFI boot), and a ZFS partition. As root:

# Clean that MBR and create GPT partition table
gpart destroy -F da2
gpart create -s GPT da2

# If you use MBR boot
gpart add -s 64K -t freebsd-boot da2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2

# If you use EFI boot
gpart add -s 128M -t efi da2
gpart bootcode -p /boot/boot1.efifat -i 1 da2

# Create the ZFS partition
gpart add -t freebsd-zfs da2

Now it’s time to replace the device:

$ zpool replace -f freenas-boot da1p2 da2p2
cannot replace da1p2 with da2p2: device is too small
# OOPS! :(

If you got the message device is too small, that is because you’ve been trying to replace a device with another one that is smaller than the original (in term of ZFS partition size I suppose). ZFS requires that you replace a device with another with a size greater or equal to the old one. This can be a problem especially if you have a ZFS partition along with others in the partition table like here, or if the vendor announces a certain disk size but only within a certain approximation. You could result in a disk announced to be the exact same size ending up to be very slighty shorter than your original disk. And as a result you won’t be able to use it to replace any disk in your ZFS pool/vdev. That’s why some recommend to create ZFS filesystems several percents less than the available disk size.

But since we messed up, instead we will create a pool on the other USB key and use ZFS send/recv to migrate the files.

# Create the pool on the new USB key.
# Note that we have to call it freenas-boot2
# because freenas-boot already exists.
zpool create freenas-boot2 da2p2 

# Create a recursive snapshot of the original pool
# that we will use to restore the backup. 
zfs snapshot -r freenas-boot@restore

# Send the snapshot from the old to the new USB key.
zfs send -R freenas-boot@restore | zfs recv -F freenas-boot2

We also need to save the original setting for bootfs because FreeNAS has its own way of managing the boot pool. We will set this value later on the new pool.

$ zpool get bootfs
NAME          PROPERTY  VALUE                        SOURCE
freenas-boot  bootfs    freenas-boot/ROOT/11.3-U4.1  local

We still have to change that name from freenas-boot2 to freenas-boot. However we cannot do so on the running instance of FreeNAS because it already has a pool named freenas-boot. So, we will import it on another ZFS capable machine and fix its name.

(but if anyone knows how to do this live, I’m interested)

# On some other ZFS capable host (FreeBSD for instance ;) )
# import the pool and change its name back to freenas-boot.
# The -f is required because it comes from another host.
zpool import -f freenas-boot2 freenas-boot

# Configure default boot filesystem.
# Remember the value of bootfs we found before on the original pool.
zpool set bootfs=freenas-boot/ROOT/11.3-U4.1 freenas-boot

# Export the pool again.
zpool export freenas-boot

Finally you can clear the old snapshot:

zfs destroy -R freebsd-boot@restore

Faster Intel graphics on FreeBSD

If you have a laptop with an Intel graphic card and architecture above Sandy Bridge (circa 2011), you can accelerate xorg significantly with this little hack. Create the file /usr/local/etc/X11/xorg.conf.d/intel.conf:

Section "Device"
  Identifier "Intel Graphics"
  Driver     "Intel"
  Option     "AccelMethod" "sna"
EndSection

See the manpage intel.4. It seems that the version of xf86-video-intel on FreeBSD falls back to the UXA acceleration method but forcing SNA provided good results on my ThinkPad X250.

Beid card reader on FreeBSD

So I’ve got to fill in my taxes, and to do this in Belgium, we need to use our ID card. Well we can use a phone app too, but to use this app we have to register… using our ID card… ¯\_(ツ)_/¯

Last time I tested, this didn’t work well on FreeBSD, but I didn’t look too much into it and perhaps things changed since then. Also, like always, I refuse to do this on another OS where I know it works, that would be far too easy!

So we should install the PC/SC-Lite architecture to be able to read the cards and the beid middleware so we can use the card. Fortunately, beid is in the ports so we can just install it:

pkg install beid

This will also install pcsc-lite. The package messages tell us that we must:

  • Install drivers for the card reader
  • Hack around devd to let the PC/SC Smart Card daemon recognize it
  • Mount procfs
  • Install the eID Belgium extension if you use Firefox (I’ll also explain how to install it in Chromium)
  • Don’t trust your government

Let’s do just all of that!

Install drivers for the card reader

My card reader is a DunnoWhat-Random-USB-Card-Reader, perhaps devel/libccid should do? For some reason, this is not available in the packages but only via ports:

cd /usr/ports/devel/libccid
make install clean

Does the card reader read cards?

Does it work tho? Let’s find out!

$ usbconfig
...
ugen0.6: <SCM Microsystems Inc. SCR35xx v2.0 USB SC Reader> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
...

So our card reader is on the USB port ugen0.6. We run pcscd to see if it’s detected properly:

$ pcscd --info --color --foreground
...
00000155 [34366794240] ccid_usb.c:660:OpenUSBByName() Found Vendor/Product: 04E6/5410 (SCM Microsystems Inc. SCR 355)
...

Sweet, so it appears to be detected and working with CCID. However if I run eid-viewer, it doesn’t work. Looking at pcscd info, it looks like the power up of the card failed:

...
00000011 [34375102464] ifdhandler.c:1221:IFDHPowerICC() PowerUp failed
...

Searching a bit over the internet, I’ve found this bug report and it seems that we have to tweak the CCID driver a little.

Edit /usr/local/lib/pcsc/drivers/ifd-ccid.bundle/Contents/Info.plist and search for <key>ifdDriverOptions</key>. By default it is set to 0x0000 which, if you read the description belows, means that it tries to power on the card at 5V, then 1.8V then 3V. But apparently this doesn’t work with my particular card reader. Instead I let the card reader decide, change the value to 0x0030:

  <key>ifdDriverOptions</key>
  <string>0x0030</string>

Restart pcscd (pcscd --info --color --foreground) and try the card again (eid-viewer). In my case, I can see my card details, so I believe this means that the card reader is working well.

Hack around devd

First, let’s enable pcscd at boot in /etc/rc.conf with pcscd_enable="YES".

Now we need to configure devd as described by the pcsc-lite package message (pkg info -D pcsc-lite). Add this to /etc/devd.conf:

attach 100 {
        device-name "ugen[0-9]+";
        action "/usr/local/sbin/pcscd -H";
};

detach 100 {
        device-name "ugen[0-9]+";
        action "/usr/local/sbin/pcscd -H";
};

The -H option is to re-read some configuration files in case of a non-USB reader. You can probably remove them.

Now let’s restart everything we need and check that it still works:

$ service devd restart
$ service pcscd restart

Check that it works with the eid-viewer.

Mount procfs

It’s probable that you already have procfs mounted because it’s needed by a lot of other ports. But if you didn’t, just add this line to /etc/fstab

proc /proc procfs rw,late 0 0

and then just mount procfs.

Install the eID Belgium extension (for Firefox)

If you use Firefox, you are looking for this extension. It should work after you have restarted Firefox.

Get it to work in Chromium

On Chromium there is no extension, instead we work directly with NSS. For beid, the process is straightforward since they provide a script to install the necessary module in NSS. Go in your home directory and start the command beid-update-nssdb. Then restart chromium completely and it should work.

Alternatively you can adapt this post which explains how to manually update nssdb on Ubuntu. You would have to adapt this to FreeBSD and also use this library /usr/local/lib/libbeidpkcs11.so.0 instead of libcac.

Don’t trust your government

You do this.

USB printer in FreeNAS jail

Running a printer in a FreeNAS jail may be interesting for home users, especially if you have one of those GDI printers that only provide a x86/amd64 driver for Linux/FreeBSD, and more especially if you happen to have a FreeNAS box in your living room with some free USB ports.

While exploring the subject, I’ve come across several solutions to use FreeNAS as a printer server. Some explain how you could install the printer inside (a very old version of) FreeNAS itself, which you should seriously not do. Not only all your changes would be overwritten by updates, but it offers limited flexibility and very little guarantee that anything will work or keep working in the future. Even if FreeNAS is FreeBSD at its core, it’s not supposed to be used as a full-fledged OS, if you want that, you should use a VM or a jail. Other solutions explain how to install the printer inside a jail but generally expose all the devices inside the jail just to access the printer. I’ve even heard of someone giving up half way to the solution presented here only to deploy a Windows VM in FreeNAS just to get the printer running.

Part 1: USB inside the jail

The main challenge will be to have the USB printer to appear inside the jail. That is for the device file to appear inside the jail /dev directory. By default, the content of this directory is pretty minimalist.

On FreeBSD (and FreeNAS), the creation of device files in /dev is handled by devfs. Each devfs mount has an associated ruleset number specifying how devices must be created on this mount-point.

FreeNAS uses iocage to manage its jails. You can get the ruleset associated to the jail devfs mount with the following command:

$ iocage get devfs_ruleset {myjail}
5

By default, this is ruleset number 5. You can list the rule in this ruleset with the command devfs rule -s 5 show, see how it matches devices that you find under your jails.

We will create a custom ruleset for our jail that also include the USB device of the printer. The script will even detect the USB port automatically. But first let’s check if we see the printer and its name:

$ usbconfig
...
ugen0.3: <Kyocera FS-1041> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (2mA)
...

We will create a new ruleset number 1000 that includes ruleset number 5 and unhides the necessary device for the jail. Also USB devices need to be owned by the group cups and permission 660. Normally, the cups port/package adds a file in /usr/local/etc/devd to change the group owner and permission when a new USB device is plugged in. However devd doesn’t work in jails, so we will also have to change these using devfs.

On a normal FreeBSD system, we could setup this ruleset in /etc/devfs.rules, however this file would be rewritten in FreeNAS on each reboot/update. Instead we create a script that will be started on each boot. We can also use this script to detect the USB port on which the printer is connected. In my case I store this script in my home directory on its own dataset (so it won’t be overwritten on reboot), but you can use any of your datasets. For this example the script will be in ~myuser/myjail-devfs-ruleset.sh.

Let’s edit that with nano:

#!/bin/sh
# Custom ruleset for jails

export PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin"

RULESET=1000
CUPS_GID=193
PRINTER_NAME="Kyocera FS-1041"

# Find the printer device.
UGEN_DEV=$(usbconfig | grep "$PRINTER_NAME" | cut -d':' -f 1)
USB_DEV=$(readlink /dev/$UGEN_DEV)

if [ -z "$UGEN_DEV" -o -z "$USB_DEV" ]
then
  echo "error: cannot find printer '$PRINTER_NAME'"
  echo "error: please check with usbconfig"
  exit 1
fi

echo "Found $PRINTER_NAME on $UGEN_DEV"

# Clean the ruleset
devfs rule -s $RULESET delset

# Include jails default ruleset and unhide USB device.
devfs rule -s $RULESET add include 5
devfs rule -s $RULESET add path usb unhide
devfs rule -s $RULESET add path $USB_DEV mode 660 group $CUPS_GID unhide
devfs rule -s $RULESET add path $UGEN_DEV mode 660 group $CUPS_GID unhide
devfs rule -s $RULESET add path usbctl mode 644 unhide

In this script you should edit PRINTER_NAME with the name of the printer found with usbconfig. You should also edit CUPS_GID with the gid of the cups group inside your jail (generally this is 193) and edit the RULESET number if needed.

Now adjust the permission and execute this script. We will also check that the rules are applied correctly:

$ chmod a+rx ~myuser/myjail-devfs-ruleset.sh
$ sh ~myuser/myjail-devfs-ruleset.sh
Found Kyocera FS-1041 on ugen0.3
$ devfs rule -s 1000 show
100 include 5
200 path usb unhide
300 path usb/0.3.0 unhide group 193 mode 660
400 path ugen0.3 unhide group 193 mode 660
500 path usbctl unhide mode 644

As you can see, the ruleset has been created correctly. We will now change the devfs ruleset used for the jail. In FreeNAS you can either do that in the GUI in Jails > myjail > EDIT > Jail Properties > devfs_ruleset or use the command line. Note that the jail has to be stopped before you can do the modifications. Let’s use the command line in our case:

$ iocage stop myjail
* Stopping myjail
  + Executing prestop OK
  + Stopping services OK
  + Tearing down VNET OK
  + Removing devfs_ruleset: 5 OK
  + Removing jail process OK
  + Executing poststop OK
$ iocage set devfs_ruleset=1000 myjail
$ iocage start myjail
* Starting myjail
  + Started OK
  + Using devfs_ruleset: 1000
  + Configuring VNET OK
  + Using IP options: vnet
  + Starting services OK
  + Executing poststart OK

Login into your jail and check that /dev/ugen0.3 and /dev/usb/0.3.0 are accessible and owned by cups. Now we will make this configuration persistent across reboot. Iocage jails have a exec_prestart property executed before applying the devfs ruleset. However if iocage detects that the devfs ruleset does not exist, it will fall back to a default ruleset. In our case, this means that iocage will always revert to this instead of using our ruleset. Instead we could execute the script as an init task in FreeNAS GUI. However it seems that when a jail is onfigured with Auto-Start, it is started before the Pre-Init tasks. Therefore we configure devfs and then start the jail manually. Create another file (for example in ~myuser/start-jails.sh):

#!/bin/sh
# Manually start jails.

sh ~myuser/myjail-devfs-ruleset.sh
/usr/local/bin/iocage start myjail

Go into Tasks > Init/Shutdown Scripts and ADD one with those info:

Type: Script
Script: ~myuser/start-jails.sh
When: Post Init
Enabled: Yes
Timeout: 10

Another problem is that iocage removes the configured devfs ruleset when the jail is stopped. So if you restart after that, the configured ruleset would not exist and iocage would switch to a default empty one that would expose all your devices in the jail. To fix that, we configure a property to also execute our script when the jail stops:

$ iocage set exec_poststop=$(ls ~myuser/myjail-devfs-ruleset.sh) myjail
exec_poststop: /usr/bin/true -> ~myuser/myjail-devfs-ruleset.sh

Finally go into the FreeNAS GUI and disable Auto-Start (Jails > myjail > EDIT > Auto-start: off).

Note that none of these shenanigans would be necessary if iocage did not fall back on another ruleset if the one specifed in the devfs_ruleset property did not exist. But unfortunately, it doesn’t. If you have a nice idea on how we can get around this problem, please feel free to comment below!

You are done with this part. The jail is now ready with the USB device for the printer (and no more than that) accessible in its dev directory.

Part 2: Install the printer on CUPS

Since this is a post about running a printer in a jail, there must be a step where we install CUPS and the printer itself within the jail. For anyone that has already installed printers on *BSD or Linux, this should be pretty straightforward. I will not go into the details and assume that the printer is compatible with the default CUPS filters.

Let’s install cups. I do so via ports to remove some of the defaults options. Otherwise cups and cups-filters would depend on wayland avahi and a lot other stuff. If you are fine with that though, you can just pkg install cups cups-filters. Via the ports, it’s a bit more involved:

$ cd /usr/ports/print/cups
$ make install clean
(remove AVAHI and DBUS)
...

# Install some dependencies as packages
# so you don't have to compile them from sources.
# Some would say you should not do that.
# But I don't.
$ pkg install ghostscript9-agpl-base qpdf
...

# Install cups-filters.
$ cd /usr/ports/print/cups-filters
$ make install clean
(remove COLORD, AVAHI, LDAP)
(select PSGHOSTSCRIPT as default PDF-to-PS renderer 
 since POPPLER would depends on wayland too)

# Lock the ports to avoid updates by pkg.
pkg lock cups cups-filters

Add cupsd_enable="YES" in /etc/rc.conf to start cups on boot. Start it with service cupsd start. By default, cups is only accessible on https://localhost:631, we will use a ssh tunnel to access it and configure the printer:

ssh -NL 10631:localhost:631 {myjail-ip-or-hostname}

Use your web browser and go to https://localhost:10631 and you should see CUPS. If you don’t have a root password on your jail (and only use sudo), it might be a good idea to setup one with passwd, even if only temporary while you install the printer.

If you have a GDI printer that needs a special binary filter that only works on Linux, you should load the linux and linux64 modules in FreeNAS and install linux-c7 in the jail:

###############################################
# Only if you need a Linux only binary filter #
###############################################

# Add to ~myuser/myjail-devfs-ruleset.sh
kldload linux
kldload linux64

# In the jail:
pkg install linux-c7

Go into Administration > Add Printer, it will ask for credentials, enter root and your root user password (don’t worry you only need to do this to install the printer). Hopefully you should see your printer in local printer.

Select it and continue. Configure name, description and location. Enable the Share checkbox, since you probably want to share this printer on your network. Continue, and select the appropriate printer model then Add printer. Set the default options.

You will be redirected to https://localhost:631 but this won’t work through the ssh tunnel (with local port 10631) so do not worry if the browser says that the site is not reachable. Just retype the URL and go to the printers https://localhost:10631/printers, select it, then Maintenance > Print Test Page.
If the CUPS Printer Test Page comes out of the printer at this point, congratulation! You’ve successfully configured a USB printer with CUPS inside a FreeNAS jail!

If on the contrary nothing comes out, remain calm, /var/log/cups/error_log might be of some use.

Part 3: Share CUPS with your network

We still have to make cups available on the network. To do so, edit /usr/local/etc/cups/cupsd.conf. You can find an example of this configuration in this CUPSD configuration example. In this example, we let everyone on the local network browse the CUPS server and create print-job. The job owner can cancel its own job and all other administrative tasks and printer manipulation are restricted to localhost. For those later operations, you should connect to cups using the ssh tunnel method presented above.

On each client (assuming that it has cups installed), you can configure /etc/cups/client.conf (Linux) or /usr/local/etc/cups/client.conf (FreeBSD) with the hostname of your jail:

ServerName {hostname-or-IP-of-CUPS-jail}

The printer should now be available. Check that lpstat works without error. If you have a problem, again have a look at /var/log/cups/error_log in the jail, it should give you a good start for debugging.

Part 4: Profitsssss

Profitsssss

Great profitssssss

[1] Is it possible to access a USB printer from inside a jail?
[2] Enable sound inside jail
[3] Absolute FreeBSD, 3rd Edition: The Complete Guide to FreeBSD
[4] Exposing Device Files to FreeBSD Jails
[5] Ruleset exists but iocage does not find it #952
[6] USB (Z-Wave) device no longer shows up in iocage jail on FreeNAS 11.2
[7] USB device inside a linux jail (devfs)

UPDATE:

  • It seems that with TrueNAS 12 the DevFS ruleset 5 does not exist anymore. As far as I remember these rules did hide everything then unhide the minimum necessary for a working jail. Now when creating the jail DevFS ruleset, you have to add the following rules:
    • hide
    • include 2
    • include 3

Kyocera FS-1040/FS-1041

Printers, who needs them anyway? You can probably just read from a PDF and as for those documents that must be printed on paper, we always have access to a printer somewhere, don’t we? Not so much anymore. So in all this lockdown situation maybe you thought about buying a budget laser printer. Maybe a Kyocera, because that’s the first LaserJet printer you owned, you remember that back in the days it was rock solid and worked flawlessly on Linux with CUPS. Surely now they should only be better, cheaper and more compatible. So you buy this new printer and find out that it’s not compatible with anything else than Windows. Congratulation, you’ve just been struck by the curse of the GDI printers. Those Windows-only printers are getting more and more common (they are cheaper to make) and it’s difficult to get them running as each model generally requires it’s own driver. So most of the time you depends on the goodwill of the manufacturer and when it does provide a driver for Linux, it’s more often than not in the form of a binary.

This has become a plague. When my parents bought a new printer several years ago, I remember asking the vendor multiple times if it was compatible with Linux and if this was a PCL-compatible printer. He assured me that, “yes, it is”, only to find out when I came home that “no, it isn’t”.

So hold on to your butts as I have a semi-great announcement to make with regards to printing with the low budget GDI printer Kyocera FS-1040/FS-1041. It just works! Not only on Linux, but also on FreeBSD. But that’s with a binary, and only on x86/amd64. That’s why it’s just semi-great news.

The great news is, it might be possible to run it with a reverse engineered CUPS filter. That means among other things that you could run this printer from an ARM platform such as a RPi. In the coming days, I plan to dedicate some of my spare time to try to get this open CUPS filter to run inside a jail on FreeNAS.

I’ll update this post as I progress and write further posts once it’s ready.

UPDATE

  • USB printer in FreeNAS jail: It works but can still be improved. Especially the problem of applying devfs ruleset to iocage jails.
  • Open KPSL filter: I got this version to work inside a FreeNAS jail. Had to hack the CMake rules a bit, but if you manage to compile it, you can replace the binary filter with it.

[1] GDI printers
[2] Reverse engineered Kyocera rastertokpsl filter for CUPS
[3] Access USB Serial Device In Jail

Mktemp with suffix/extension

If you want to create a unique temporary file in a shell script, you would use the mktemp command. You can even specify a template where XXXXXX would be replaced by a unique combination. For instance:

mktemp ./my-temporary-file.XXXXXX
./my-temporary-file.j2iuMR

Now what happen if you want to create a temporary file ending with some particular suffix? You may want to do that because the program to which you feed your temp file expects some file extension to parse it properly. For example most web browsers expect files ending in .htm or .html to parse them as HTML documents. However, if you try to provide mktemp with a template ending with the appropriate suffix, that wouldn’t work (at least not on FreeBSD and OpenBSD):

mktemp /tmp/tmpXXXXXX.html
/tmp/tmpXXXXXX.html
# AAAARG! THIS IS NOT VERY UNIQUE! x_x

The version of mktemp in GNU coreutils comes with a --suffix option that allows you to do just that. But this is specific to GNU so you should not use that if you care about your scripts and other people. And please, do care about other people. Truly, scripts expecting /bin/sh to be bash or some other Linuxism are a real chore to work with, even if you work on Linux. So please restrain yourself and do the right thing.

A first solution that would come to mind would be to create the temporary file and move it to the same name but with the appropriate suffix, like this:

# Don't do that it's wrong!
tmp=$(mktemp)
tmp_html="${tmp}.html"
mv "$tmp" "$tmp_html"

But this is wrong! So don’t do that. The problem is that when you move your file, you don’t know if another file with the name "$tmp_html" is already present. It may be very unlikely, but not impossible. You may want to check if the file exists before executing the move but you could never completely avoid a potential race condition that mktemp was supposed to fix.

So a more correct answer is to create a temporary directory, and create your file in it:

tmp_d=$(mktemp -d)
tmp_f="$tmp/myfile.html"

... do your things ...

rm -r "$tmp_d"

With this, you know that your directory is unique, and as long as you are the only person using it, any file created in it should be unique too.

Hexadecimal Editors

Real editors are hexadecimal editors. Over the years I’ve come to use some of them on various platform. But when it comes to use it on my everyday computer, that’s almost always inside some other tool integrating an hex view/editor like Wireshark, tcpdump or some debugger. Still, there are times when you need some standalone hex viewer/editor on a raw file and when it comes to this, it’s always hard to decide between the gazillion ones you have tried in the past. So I’ll try to make a short list here, not only to help you decide, but also to help me remember.

HT editor

I’ll start with one of my long time favorite. You only have to look at the website to see how cool and old school it is. The interface reminds me of QBasic or Edit on MSDOS. Not only does it serve as a very hex/text editor, it also has a raw ASM (x86, AMD64, IA-64, Alpha, PowerPC, ARM) mode and PE/ELF mode where you can view/edit headers and a full fledged disassembler.

hexyl

hexyl

A cool replacement for the hd command. Displays hex with more fancy and colors.

hexedit

hexedit

A very simple and clear command line interface, nothing more, nothing less. F1 for help, Ctrl+X to exit, Tab to switch between hex/ASCII edit. That’s all you need to know. I regret that you cannot directly create a file while opening it, that could be useful to craft your own packet payload for instance. Instead you need to create the file before hand and then edit it.

hexedit (2)

Another one called hexedit, on FreeBSD this is chexedit. The nano of the hexadecimal editors, a very simple curses interface. You have to start it with -b to enable the insert mode. Also it won’t create files that don’t exist or edit empty file (it has to be at least 1 Byte).

Ghex

Ghex

The GNOME hexadecimal editor. True to the GNOME interface, it’s very clear and simple. Similar to hexedit but with some added features thanks to the GUI. It’s pretty basic tho.

wxHex Editor

wxHex Editor

Has some additional features compared to GHex. In addition to the search features you find in most GUI hex editor, there is also a raw disassembler based on selection, and you can compare binary files in hexa.

Hexer

Hexer

The vim-like version of the hex editor, still very simple and clean like hexedit.

Okteta

Okteta

The KDE touch to the hexadecimal editor. It has some useful tools that you can use for further analysis like a structure decoder, frequency analysis, string extractor, bookmarks table. I just wish it was more easy to configure the structure decoder on the fly.

Beyond dual boot, make Linux and FreeBSD friends

This is a continuation of a previous post on dual boot UEFI Linux/FreeBSD.

Dual boot is not enough and we want both FreeBSD and Linux to get along for the better. To this end we will discuss several aspect of making both OS recognize each other and work in parallel.

Note that this post in particular is subject to changes. These things evolve rapidly so some details may not be accurate, not up-to-date or differ from your specific setup. So please feel free to comment/update/add ideas/point out errors or missing details. Thanks!

We will look at the following aspect:

Access each other data

The first step is to recall your partition layout and what it looks like on FreeBSD and Linux. Take time to note it on a piece of paper. That’s especially useful if you have extra hard disks with additional data partitions. To recall in the previous post we had:

Name FreeBSD Linux
EFI /dev/ada0p1 /dev/sda1
Linux SWAP /dev/ada0p2 /dev/sda2
Linux ‘/’ (ext4) /dev/ada0p3 /dev/sda3
FreeBSD SWAP /dev/ada0p4 /dev/sda4
FreeBSD ‘/’ (UFS) /dev/ada0p5 /dev/sda5

FreeBSD and Linux can both access their counterpart ext4 and UFS partition. In particular, FreeBSD can also write to Linux’s ext4 partitions. Linux supports writing to UFS in theory, but I would strongly recommend against that. Last time I tested, it only completely wrecked the UFS partition and had to reformat it. In fact, I’d recommend that you mount the counterpart OS partition in read-only to avoid messing with anything. Eventually setup another shared ext4 partition that you access in writing from both Linux and FreeBSD. At least if something goes wrong, you only loose that.

As for the swap partitions, both OS can use each other partition. Linux however requires a special signature which can be created with mkswap. However I don’t think this is still required and it works fine without it.

We will add the mount point in /mnt, that’s as good a place as any. Of course everything below must be done as root.

FreeBSD

Create the mount point for Linux:

mkdir /mnt/linux

We want to mount the second (Linux swap) and third (Linux ext4) partitions, that is for FreeBSD /dev/ada0p2 and /dev/ada0p3. Add the partitions to the end of /etc/fstab:

/dev/ada0p2 none swap sw 0 0
/dev/ada0p3 /mnt/linux ext2fs failok,ro 0 0

Notice the failok option for /mnt/linux. It means that the FreeBSD boot should not fail if the ext4 partition fails to mount. Otherwise, if the partition was marked as dirty by Linux (for instance you did a hard reboot) and then you reboot directly into FreeBSD, mounting the ext4 partition would fail because it requires an fsck.

Linux

Create the mount point for FreeBSD:

mkdir /mnt/freebsd

We want to mount the fourth (FreeBSD swap) and fifth (FreeBSD UFS) partitions, that is for Linux /dev/sda4 and /dev/sda5. Add the partitions to the end of /etc/fstab:

/dev/sda4 none swap sw 0 0
/dev/sda5 /mnt/freebsd ufs nofail,ro,ufstype=ufs2 0 0

Notice the nofail similar to failok in FreeBSD.

Share documents

Now that both OS can access each other data, it’s time to see if we can put this to some use. A first thing that you might do is to share some parts of your home directory. Of course this will be read-only.

What I personally do is selecting the OS that I use most frequently. This one contains everything. Then on the other OS, I use symlinks to some part of my home directory. For instance on Linux:

  • /home/user/Music    -> /mnt/freebsd/home/user/Music
  • /home/user/Pictures -> /mnt/freebsd/home/user/Pictures
  • /home/user/Videos   -> /mnt/freebsd/home/user/Videos

Those don’t change that often when I’m on Linux, but I least I got to listen to music, watch movies and can access pictures.

Share ssh keys

If you frequently use ssh to access your laptop or rsync+ssh to sync your documents, you will soon find yourself with ssh complaining that the host key has changed on the same host. Of course FreeBSD and Linux will both have a separate set of ssh host keys. Thankfully we can use the same key on both.

Suppose that we use FreeBSD’s ssh host keys. On Linux, go into /etc/ssh:

# Remove Linux's ssh host keys
rm ssh_host_*key*

# Link FreeBSD's ssh host keys
for key in /mnt/freebsd/etc/ssh/ssh_host_*key*
do
  ln -s "$key"
done

Don’t forget to service ssh restart. Now you can access both Linux and FreeBSD with ssh as if it was the same host (which it is actually).

Share WPA supplicant (WiFi)

Same principle now for WPA supplicant configuration file. However it’s not as simple as it was for ssh. You see, the WPA supplicant configuration needs different options for the control socket on FreeBSD and Linux. Unfortunately, wpa_supplicant.conf does not allow for file include. Actually it should be possible to arrange FreeBSD and Linux so that the same wpa_supplicant.conf is used on both OS. But this option allows for more flexibility. So here is what I do.

First I create /etc/wpa in both FreeBSD and Linux. Then I edit /etc/wpa/local.conf with the wpa_supplicant options specific to this OS. Then I use this small script to select a particular profile and create the appropriate wpa_supplicant.conf.

#!/bin/sh

if [ ! -r "$1" ]
then
  echo "error: cannot read '$1'"
  exit 1
fi

cat /etc/wpa/local.conf > /etc/wpa_supplicant.conf
cat "$1" >> /etc/wpa_supplicant.conf

The idea behind those profiles is to restrict scanning of new networks depending on the situation. For instance you can have one profile for your home, one for your working place, one when you go abroad. It’s easier to organize your configuration that way and also avoids to send probe requests on the air that can disclose information about you.

Thus on FreeBSD, I create /etc/wpa/profiles along with the various profiles and on Linux, I just link to it.

Share nullmailer

If you happen to use nullmailer as your local MTA, you can share your smtp credentials too. But there is a catch. The remotes file in the nullmail configuration must be owned by the nullmailer user/group. This user/group is different on Linux than it is on FreeBSD (nullmail on FreeBSD, mail on Linux). Fortunately when you mount a filesystem, it only cares about the UID/GID, not the actual user/group name. So if we change the UID/GID of the user/group mail to match the UID/GID of the nullmail user/group on FreeBSD, it will appear as the same user but with a different name on each OS. That’s what we’ll do on Linux, we will change the UID/GID of mail to match the one on FreeBSD.

First, let’s list all files owned by user mail:

find / -xdev -user mail > user
find / -xdev -group mail > group

Second, check that we won’t mess around too much by changing these files:

$ cat user group | sort | uniq
/etc/nullmailer/remotes
/usr/bin/dotlockfile
/usr/bin/mailq
/usr/sbin/nullmailer-queue
/var/mail
/var/spool/nullmailer
/var/spool/nullmailer/failed
/var/spool/nullmailer/tmp
/var/spool/nullmailer/trigger

Seems OK. Now let’s find out the UID/GID of the nullmail user group on FreeBSD:

$ cat /mnt/freebsd/etc/passwd | grep nullmail
nullmail:*:522:522:Nullmailer Mail System:/var/spool/nullmailer:/bin/sh
$ cat /mnt/freebsd/etc/group | grep nullmail
nullmail:*:522:

The UID/GID is 522:522 on FreeBSD. We will change user/group mail on Linux to UID/GID 522:

# Stop nullmailer so that we can change the UID/GID
service nullmailer stop

# Change UID/GID
usermod -u 522 mail
groupmod -u 522 mail

# The files are still owned by the old UID/GID.
# We change that
cat user | while read file
do
  chown mail: "$file"
done
cat group | while read file
do
  chown :mail "$file"
done

# Clean
rm user group

Now we link nullmailer configuration into Linux:

rm -rf /etc/nullmailer
ln -s /mnt/freebsd/usr/local/etc/nullmailer /etc/nullmailer

You should be able to service nullmailer restart now.

Dual boot UEFI Linux/FreeBSD

There a lot of tutorials out there explaining how to dual boot Linux and FreeBSD on legacy BIOS but not so much for UEFI only systems. So I will share my experience installing Debian 10.2 and FreeBSD 12.1 on my ThinkPad X250 in UEFI only mode. It should be easy to adapt this to other Linux distributions and other systems than FreeBSD.

This post will be soon followed by another post explaining how FreeBSD and Linux can get along with each other after being installed. But now for the installation.

First ensure in your UEFI/BIOS settings that boot is set to UEFI only and CSM disabled. You don’t want to boot the installer in legacy mode by accident.

We will use a GTP partition table and the create the following partitions:

  1. EFI System Partition (ESP): To store the UEFI bootloaders
  2. SWAP + ext4: For Linux
  3. SWAP + UFS: For FreeBSD

Note that we could technically use the same SWAP for both FreeBSD and Linux. Still I prefer to use 2 SWAP partitions and use both of them in each OS. It’s a matter of preference I guess.

Install Linux and FreeBSD

Install Linux

We start installing Debian so that Linux stays in charge of GRUB. We do so because it’s frequent on Linux to have multiple kernel variants, for instance a more recent version of the kernel, custom or a RT patched kernel. So we let Linux’s package manager handle all of that.

At the partitioning step, select “Manual partitioning” and create a GPT partition table. For UEFI to function properly, we need a EFI System Partition (ESP), 500MB should be more than enough for this. Then a Linux swap partition and a Linux ext4 partition for the data. Leave some space unallocated for FreeBSD.

It is possible to use Linux’s swap partition in FreeBSD. More about that later. For now we will let each OS have its own swap partition.

Complete the Debian installation. It should install grub bootloader in the EFI partition. Check that Debian boots properly. Then start the FreeBSD install.

Install FreeBSD

There used to be a separate installation image for FreeBSD UEFI. This is not the case anymore, so you can use the AMD64 memstick image for 12.1-RELEASE on the FreeBSD download page.

Boot the installer and go ahead up to the partitioning step. Use the space you left unallocated for the freebsd-swap and freebsd-ufs partitions. The installer will complain that an EFI partition is required for the system to work properly and propose to create it. Ignore this as the partition was already created under Linux. It is weird though that the FreeBSD installer does not detect this, but there is a FreeBSD forum post about this issue.

Proceed and complete the FreeBSD installation. Then reboot into Debian to configure the dual boot.

GRUB dual boot

It is now time to tell Linux’s GRUB about our newly installed FreeBSD system.

We created the swap partitions before the data partitions for each OS, so to resume our partition table we now have:

  1. EFI System Partition
  2. Linux SWAP
  3. Linux ext4
  4. FreeBSD SWAP
  5. FreeBSD UFS

So our FreeBSD partition is (hd0, gpt5) in GRUB parlance. You may need to adapt this to your own partition scheme though. Once in Debian as root edit /etc/grub.d/40_custom and after the comment add:

menuentry 'FreeBSD' {
insmod ufs2
set root='(hd0,gpt5)'
chainloader /boot/loader.efi
}

If you want, you can also configure FreeBSD as the default entry by editing /etc/default/grub and change DEFAULT=0 to DEFAULT=FreeBSD.

Then update grub with update-grub2, finally reboot and select FreeBSD in the GRUB menu. You can now boot both Linux and FreeBSD.

FreeBSD aware UEFI

It is now possible to boot both Debian and FreeBSD from GRUB. However it is not yet possible to boot FreeBSD directly from UEFI. To do so we need to copy the FreeBSD UEFI loader in to EFI partition and register it. Debian already mounts the EFI partition but FreeBSD doesn’t, so for the fun of it, let’s manage all that under FreeBSD and install the FreeBSD UEFI loader. /dev/ada0p1 is the EFI partition, but you may need to adapt this to your partition scheme though.

# We mount the EFI partition on /boot/efi similarly to Linux.
mkdir /boot/efi
echo '/dev/ada0p1 /boot/efi msdosfs rw,noatime 0 0' >> /etc/fstab
mount /boot/efi

# Install the FreeBSD UEFI loader.
mkdir /boot/efi/EFI/freebsd
cp /boot/boot1.efi /boot/efi/EFI/freebsd/bootx64.efi

Now let’s create an UEFI entry for this loader. Note that this is for FreeBSD’s efibootmgr, not the Linux’s one.

# Create the boot variable.
efibootmgr -c -l /boot/efi/EFI/freebsd/bootx64.efi -L "FreeBSD"

# Check the variable number for the new boot variable and activate it.
efibootmgr
efibootmgr -a 15

# Change the boot order to leave Debian and GRUB in charge.
efibootmgr -o 14,15

Time to reboot! Select the boot menu with (generally with F12, at least on my ThinkPad X250) and FreeBSD should appear. Select it and it should boot FreeBSD directly.

You are done! Next time how to let FreeBSD and Linux talk to each other.