Bhyve OpenBSD on FreeNAS

Lately I’ve been playing with bhyve on FreeBSD and FreeNAS in the prospect of spinning up small VMs for a mini compile farm (and just as an excuse to play around). In this post I will share my experience in installing OpenBSD 6.4 as a bhyve UEFI VM on FreeNAS-11.2 through the new GUI.

This post is divided in three parts. First the creation of the VM through FreeNAS new GUI. Then the installation of OpenBSD from boot to finish. Lastly some bits of documentation and related posts that were helpful along the way.

Create the VM

The installation will go through serial instead of VNC. Also, instead of the CDROM ISO we use a disk image for the installation.

In the FreeNAS GUI, selects Virtual Machines > ADD, then proceed with the wizard. Some parameters of the wizard are pretty obvious, so there won’t be an explanation for each one of them. If need be, check the FreeNAS doc on creating VMs.

Guest Operating System, not exactly sure what this does. Guess it’s basically a template for the next steps of the wizard. Since this is a BSD system, select FreeBSD as this is the closest there is.

We don’t need VNC, so make sure that Enable VNC is unchecked. As for Boot Method, you can select UEFI, you don’t need UEFI-CSM.

For the hard disk in Select Disk Type and similarly for the network interface in Adapter Type, select a VirtIO type.

For the Installation Media, leave it blank. This is for an ISO image but OpenBSD’s installXX.iso didn’t work so we are going to use a raw disk image instead.

Go ahead and create the VM. But don’t start it yet. We have to add a raw disk device.

Fetch installXX.fs from OpenBSD download page, and put it somewhere on the FreeNAS host. For the following, I’ll assume this is install64.fs.

Then select Devices > ADD for the VM. Select Type: Raw File and Raw File: the location of install64.fs on the FreeNAS host. Also Mode: VirtIO; Device Order: 1003 (the device order is important, otherwise the OpenBSD install would incorrectly guess the target device); Raw filesize: 1 (which means 1GB).

Now you can start the VM and open the Serial console. You should be greeted by:

>> OpenBSD/amd64 BOOTX64 3.40
boot>

Time to install OpenBSD!

Install OpenBSD

You are on the serial console with the OpenBSD install bootloader waiting for you. On the console boot>, type set tty com0, then boot.

Do not directly start the installation, we first have to create the EFI partition, so select (S)hell instead. At the command prompt, create the EFI partition as described below, then start the install.

## Check that sd0 is the correct target device
# disklabel -p M sd0
...
16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:         15625.0M                0  unused

## Initialise a GPT partition table with the special boot partition on sd0.
# fdisk -iyg -b 960 sd0
Writing MBR at offset 0.
Writing GPT.

## Start the installer
# install

At the partitioning step, select (O)penBSD area. We will setup a single root partition layout. You should know that the OpenBSD bootloader likes its root partition on slice a of the first hard disk, so we create the layout that way.

# Check the current partition
> p M
OpenBSD area: 1024-31999937; size: 15624.5M; free: 15624.5M
#                size           offset  fstype [fsize bsize   cpg]
  c:         15625.0M                0  unused                    
  i:             0.5M               64   MSDOS 

# Start with the root partition.
# Again, the bootloader likes it that way.
> a
...
size: {your-root-partition-size}M
FS type: 4.2BSD
mount point: /

# Now the swap partition
> a
...
FS type: swap

# Check again
> p M
OpenBSD area: 1024-31999937; size: 15624.5M; free: 0.0M
#                size           offset  fstype [fsize bsize   cpg]
  a:         15493.9M             1024  4.2BSD   2048 16384     1 # /
  b:           130.5M         31732576    swap               
  c:         15625.0M                0  unused                 
  i:             0.5M               64   MSDOS

# Quit and save
> q
Write new label?: y

Proceed with the installation and the file sets and once you are done, reboot in your new system. Remove the Raw File device from the VM and on the VM itself, use syspatch to patch the base system. Finally reboot and you are done!

Relevant bits of documentation

Samba denied on FreeNAS

If you recently discovered that some of your authenticated samba shares on your FreeNAS refuse to well authenticate. And maybe you did your own investigation and found that it failed with NT_STATUS_NO_SUCH_USER. Do not worry, you are not alone!

From what I understand, the last update changed the constraints on the user accounts metadata. In particular it seems that the E-Mail field must now point to a real e-mail address (something@domain) and cannot just be an alias to another local user.

Don’t know if that’s intended or not or why it would be. Going to IRC right now to find out.

Long story short, users that don’t match the new constraints are not recognized anymore by samba (and probably other services too), hence the permission denied and NT_STATUS_NO_SUCH_USER. So for those user using a local alias as E-Mail, use a fully qualified E-Mail. That will fix the shares. However it seems that "user"@localhost doesn’t work either. And you will also have to retype their password.

Wrong OS version in pkg

If FreeBSD’s pkg refuses to update because of a wrong OS version, for example because you are in a FreeNAS jail and version mismatch are bound to happen, then you can always try to force the OS version in pkg (which you can find with uname -UK. Either put OSVERSION={{version}} in /usr/local/etc/pkg.conf or start it with pkg -o OSVERSION={{version}} where {{version}} is the target version, for example 1101001.

NAS (pt. 1): Parts and build

In a previous post, I explained how a disk failure prompted me to build that NAS I’ve dreamed of for years. That happened months ago, but in the meantime we’ve been reviewing mainboards, CPU specs, reading other configurations over and over again. We finally settled on a setup of our own.

We did so with a set of constraints. First anything with less than 4 HDD was out of the question. You need at least 3 HDD for RAID5, 4 for RAID6, with RAID1 (mirroring) and only 2 HDD you would waste half of your total disk space, and RAID0 (no redundancy) would be completely insane, disk failures do happen, believe me! Since we also want to upgrade our setup, we looked for mainboards with at least 6 SATA.

Second, it would run FreeNAS, anything else doesn’t come close. We avoided some component that are known to cause trouble in FreeBSD, mostly some NIC. Although as far as I can remember I didn’t have any problem for a while.

Finally I’ve been a long time advocate of ECC memory for NAS. However ECC memory is expensive and finding cheap and reliable mainboard/CPU that support ECC is difficult. So we relaxed that constraint. But I’d be happy to know about any good alternative.

Parts

Our base setup is 12TB HDD / 16GB RAM upgradable to 24TB HDD / 32GB RAM. Total cost was 688€ and 310€ without any HDD. We were lucky enough to receive HDD from different batches even though we bought all of them from the same vendor. Similar HDD would increase the risk of multiple disk failing at the same time (which is a very bad thing for a RAID5 setup).

  • CPU: Intel G3900
  • MB: Gigabyte GA-B150M-DS3P
  • RAM: 2x8GB Crucial DDR4
  • HDD: 3xWD Red 4TB
  • PSU: Corsair VS350
  • Case: CoolerMaster Elite 343
  • USB: SanDisk Cruzer Fit SDCZ33-016G-B35

Now if we could change only one thing, that would be the CM Elite 343 case. While it is possible to mount 6 HDD, their positioning is not optimal and arranging the cables properly was difficult.

Build

The build was relatively straight forward. As I said we had some problem with the CM Elite 343 case. Another problem was the SYS_FAN cable from the case which was too short to reach the MB. We also had some problem with one of the two mainboards which constantly rebooted. We had to reset the CMOS to fix it. But now everything works flawlessly.

I already booted FreeNAS 11, everything seems to be working properly. The idle consumption is 28W, peaks at 60W on boot, but we’ll see about that when the system will be fully installed. The setup is amazingly silent, although the front system fan is not connected for now.

In the next episode I will install and configure FreeNAS on Gandalf (it already has a name).  Until then I’m going to fold the gazillion amazon boxes spread all over my apartment.