Install FreeBSD 11 with ZFS on Dedibox XC 2016

Online.net’s Dedibox XC 2016 comes with 16 GB DDR3 and 1 To SATA or 250 GB SSD on a 8 cores Atom CPU. This is a very nice entry-level dedicated box for anyone who want to upgrade from a small VPS (yes, there is some upgrade in the air). There is only one HDD and no RAID though. But they offer (for free) a 100 GB FTP storage space which is more than enough to backup the base system and bootstrap it again in case of disk failure.

An advantage of dedicated over VPS is that you can install almost any OS you want. The management console comes with an easy install for FreeBSD 11 on UFS. But I thought it would be nice to use ZFS instead. Yeah, I hear you, why using ZFS with only one HDD and non-ECC memory? But with 16 GB it still comes as a viable alternative.

The method I used was adapted from a post on Online.net’s forum. So here we go. First, reboot in rescue mode from the console. Choose FreeBSD 10.2 (or higher) as the rescue OS. Once you are logged on the rescue, switch to root and bootstrap FreeBSD:


SWAP_SIZE=4g
TEMP_ROOT_PASSWORD="1337rul35"

# Create partitions table
gpart destroy -F ada0
gpart create -s gpt ada0
gpart add -t freebsd-boot -l boot -s 512K ada0
gpart add -t freebsd-swap -l swap -s $SWAP_SIZE -a 1m ada0
gpart add -t freebsd-zfs -l zfs0 ada0

# Install MBR
dd if=/dev/zero of=ada0p3 count=560 bs=512
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

# Create ZFS pool and FS
zpool create -f -m none -o altroot=/mnt -o cachefile=/tmp/zpool.cache -O compress=lz4 -O atime=off zroot gpt/zfs0
zfs create -o mountpoint=/ zroot/ROOT
zfs create -o mountpoint=/usr zroot/usr
zfs create -o mountpoint=/var zroot/var
zfs create -o mountpoint=/tmp zroot/tmp
zfs create -o mountpoint=/www zroot/www
zfs create -o mountpoint=/usr/home zroot/usr/home
zpool set bootfs=zroot/ROOT zroot

# Bootstrap
cd /mnt
fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.0-RELEASE/base.txz
fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.0-RELEASE/kernel.txz
tar --unlink -Jxpf base.txz -C /mnt
tar --unlink -Jxpf kernel.txz -C /mnt
rm base.txz kernel.txz

# Configuration
# 1) fstab and swap
cat << EOF > /mnt/etc/fstab
ada0p2 none swap sw 0 0
EOF

# 2) rc.conf
cat << EOF > /mnt/etc/rc.conf
keymap="fr.acc"
ifconfig_igb0="DHCP"
ifconfig_igb1="DHCP"
fsck_y_enable="YES"
background_fsck="YES"
zfs_enable="YES"
sshd_enable="YES"
EOF

# 3) loader.conf
cat << EOF > /mnt/boot/loader.conf
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot/ROOT"
boot_multicons="YES"
boot_serial="YES"
comconsole_speed="9600"
console="comconsole"
comconsole_port="0x2F8"
EOF

# 4) TTY for serial console
cat << EOF >> /mnt/etc/ttys
ttyu1 "/usr/libexec/getty std.9600" vt100 on secure
EOF

# 5) Temporary root password
echo "$TEMP_ROOT_PASSWORD" | pw -R /mnt user mod -n root -h 0

# Last step
cd ~
zpool export zroot
zpool import -o altroot=/mnt -o cachefile=/tmp/zpool.cache zroot
cp /tmp/zpool.cache /mnt/boot/zfs

# Terminated!
halt

Now from the management console, reboot in normal mode and connect to your box using serial connection. You should be able to login with root and continue the configuration from there.

Filter ICMPv6 with tcpdump

If you want to filter ICMP echo-requests with tcpdump, you can use this command:

tcpdump -i em0 "icmp[0] == 8"

But it doesn’t work if you try the same syntax with ICMPv6:

tcpdump -i em0 "icmp6[0] == 128"
tcpdump: IPv6 upper-layer protocol is not supported by proto[x]

Instead you can parse directly the IPv6 payload. An IPv6 packet is 40 bytes long, and the first 8 bits of the ICMPv6 header specify its type:

tcpdump -i eth0 "icmp6 && ip6[40] == 128"

The most common ICMPv6 types are:

  • unreachable: 1
  • too-big: 2
  • time-exceeded: 3
  • echo-request: 128
  • echo-reply: 129
  • router-solicitation: 133
  • router-advertisement: 134
  • neighbor-solicitation: 135
  • neighbor-advertisement: 136

IPv6 rDNS stub-zone on unbound

We have stub-zones configured on our gateway for reverse IPv6. Our ISP doesn’t delegate rDNS but we still want to lookup addresses (at least on the local side). To do so I configured a stub-zone from unbound, our local caching DNS, to our own rDNS authoritative server. Apparently unbound wants an IPv6 for its IPv6 rDNS queries to the stub-zone. Since IPv6 is not always working I solved that using the local interface. That is, the rDNS authoritative listen on localhost:5353 and unbound uses this as its stub-zone addresses.

On the authoritative (here NSD):

 ip-address: ::1@5353
 ip-address: 127.0.0.1@5353
 do-ip6: yes

On unbound, note that we need do-not-query-localhost: no to allow queries on localhost:

 do-not-query-localhost: no

stub-zone:
 name: 0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa.
 stub-addr: ::1@5353
 stub-addr: 127.0.0.1@5353

Back in v6

After quite a long time we are now back in the v6 world thanks to Hurricane Electric.
We lost our IPv6 connectivity when migrating our VPS from OpenVZ to KVM. There is no IPv6 on the newer OVH VPS 2016 although we had one on the older version. I don’t know why it is and when asked via a ticket they assured me that it would be available soon. This was months ago, still no IPv6, and I am not alone. This is becoming long and really awkward for OVH, supposed to be the 3rd hosting provider in the world.

Setting up the tunnel with HE was painless. You just configure a simple 6in4 (gif on BSD / sit on Linux) tunnel from your IPv4 to the endpoint they provide to you and your are done!

In the meantime we also configured the IPv6 prefix that we received from our ISP. I used ndppd, an NDP proxy daemon, so that our ISP modem believes that the IPv6 hosts are located on the same link and not one or more hop away (as they really are indeed, there is an intermediate router between our LAN and the modem). So we don’t need SixXS anymore which is great!

IPv6 representation in NSD

I use NSD as my authoritative DNS. Today I made a mistake while entering new AAAA records. Actually it was a bug in one of the scripts we use to manage our zones. Take the following representation, 2001:aaaa:bbbb:cccc::1111:2222:3333:4444, it is not a valid representation for an IPv6 address. According to the RFC4291, the double-colon symbol (::) indicates one or more groups of 16 bits of zeros. Since 8 groups of 16 bits are explicitly written in the preceding representation, the 128 bits of the IPv6 address are already detailed so there can be no extra group of zeros. The reason I made this mistake is that if you often work with /48 prefixes and you don’t care about your 65k subnets, all your IPv6 addresses will have zeros between the prefix and the interface ID.

Most programs will detect this as invalid, probably because they use inet_pton() to translate the address into its binary representation. Some others, such as ipv6calc parse the address manually and do not consider this as a buggy representation. In the latter case, the group is considered empty, and the double-colon implicitly translated to a single colon.

If you use this kind of invalid representation in a AAAA record, NSD will not reload your zone correctly, but on the other hand, neither will it complain and the queries will be made on a database that still contains your old zone. Of course this would not happen if you used nsd-checkzone to check your zone before reloading it.

Redirect traffic to loopback

Today I wanted to transparantly redirect the DNS requests coming at the output of a tunnel to a local caching DNS resolver. The caching DNS was listening only on the loopback as port 53 was already bound to other interfaces. That would be fairly simple on Linux:

echo 1 > /proc/sys/net/ipv4/ip_forward

iptables -t nat -A PREROUTING -i tun0 -p udp --dport 53 -j DNAT --to-destination 127.0.0.1
iptables -A FORWARD -i tun0 -o lo -p udp --dport 53 -j ACCEPT

But… The kernel will refuse to route packets with the loopback as source or destination because this qualify as a martian packet. The solution was to enable the route_localnet flag. As stated in the kernel documentation:

route_localnet – BOOLEAN: Do not consider loopback addresses as martian source or destination while routing. This enables the use of 127/8 for local routing purposes (default FALSE).

This is per interface. So I just had to enable this on the tunnel interface:

echo 1 > /proc/sys/net/ipv4/conf/tun0/route_localnet

FreeBSD 10 on ThinkPad X250

UPDATE:


I’m now using FreeBSD 10-RELEASE on a ThinkPad X250. It’s been quite a while now, and I should have written this long ago. Well… Here we go! Everything works fine except for:

  • Intel Broadwell integrated graphics
  • Intel Wireless 7265
  • Resume/Suspend

For the integrated graphics, the Broadwell architecture is not yet supported [1, 2]. But it works well using VESA at native resolution (1920×1080). I had to tweak the MTTR a little to get reasonable performance. Of course you cannot use your VGA and miniDP anymore. That’s a bummer for giving presentations. To this regard, I still rely on Linux. Also you cannot adjust screen luminosity, it’s always at full power. So if you are in a dark room or a car at night, you will transform yourself into a lamppost.

Here is what I have in my /etc/rc.local for the MTTRs:

memcontrol clear -b 0xc0000000 -l 0x20000000
memcontrol set -b 0xc0000000 -l 0x20000000 -o BIOS write-combine

I also had to force the DPI in /usr/local/etc/X11/xorg.conf.d/dpi.conf

Section "Monitor"
  Identifier "Monitor0"
  Option     "DPI" "96x96"
EndSection

For the WiFi, there is an iwm driver coming in FreeBSD 11 that was ported from OpenBSD. Until recently I used a Ralink RT2500USB card to get it working. But last week, I did a quick and dirty backport for FreeBSD 10, it is available here. This is a fork of the iwm driver before its inclusion in HEAD. It works somewhat. But please keep in mind that I have no clue here. I still have to read more and get into the FreeBSD kernel. And this is a temporary solution while we are waiting for the release of FreeBSD 11.

The driver still crashes when loaded with virtual box modules. Also the channel is currently locked. So if you want to use it, you need to find the correct channel first, select it manually, and then try to associate. There is a small script on the repo to do so.