NAS (pt. 1): Parts and build

In a previous post, I explained how a disk failure prompted me to build that NAS I’ve dreamed of for years. That happened months ago, but in the meantime we’ve been reviewing mainboards, CPU specs, reading other configurations over and over again. We finally settled on a setup of our own.

We did so with a set of constraints. First anything with less than 4 HDD was out of the question. You need at least 3 HDD for RAID5, 4 for RAID6, with RAID1 (mirroring) and only 2 HDD you would waste half of your total disk space, and RAID0 (no redundancy) would be completely insane, disk failures do happen, believe me! Since we also want to upgrade our setup, we looked for mainboards with at least 6 SATA.

Second, it would run FreeNAS, anything else doesn’t come close. We avoided some component that are known to cause trouble in FreeBSD, mostly some NIC. Although as far as I can remember I didn’t have any problem for a while.

Finally I’ve been a long time advocate of ECC memory for NAS. However ECC memory is expensive and finding cheap and reliable mainboard/CPU that support ECC is difficult. So we relaxed that constraint. But I’d be happy to know about any good alternative.

Parts

Our base setup is 12TB HDD / 16GB RAM upgradable to 24TB HDD / 32GB RAM. Total cost was 688€ and 310€ without any HDD. We were lucky enough to receive HDD from different batches even though we bought all of them from the same vendor. Similar HDD would increase the risk of multiple disk failing at the same time (which is a very bad thing for a RAID5 setup).

  • CPU: Intel G3900
  • MB: Gigabyte GA-B150M-DS3P
  • RAM: 2x8GB Crucial DDR4
  • HDD: 3xWD Red 4TB
  • PSU: Corsair VS350
  • Case: CoolerMaster Elite 343
  • USB: SanDisk Cruzer Fit SDCZ33-016G-B35

Now if we could change only one thing, that would be the CM Elite 343 case. While it is possible to mount 6 HDD, their positioning is not optimal and arranging the cables properly was difficult.

Build

The build was relatively straight forward. As I said we had some problem with the CM Elite 343 case. Another problem was the SYS_FAN cable from the case which was too short to reach the MB. We also had some problem with one of the two mainboards which constantly rebooted. We had to reset the CMOS to fix it. But now everything works flawlessly.

I already booted FreeNAS 11, everything seems to be working properly. The idle consumption is 28W, peaks at 60W on boot, but we’ll see about that when the system will be fully installed. The setup is amazingly silent, although the front system fan is not connected for now.

In the next episode I will install and configure FreeNAS on Gandalf (it already has a name).  Until then I’m going to fold the gazillion amazon boxes spread all over my apartment.

WIDE DHCPv6 flood

On FreeBSD we generally use WIDE DHCPv6 (also known as KAME DHCPv6, dhcp6c or simply dhcp6) as DHCPv6 client. However a rare bug can trigger this client to flood the DHCP server with requests. This happened to us and quickly prompted online.net to block our server for outgoing flood. This scared me a bit at first, as I thought we might have been part of a DDoS attack. Thankfully that was not the case.

But we still had to disable dhcp6 (and consequently IPv6). On Linux it is generally recommended to limit the DHCPv6 traffic using iptables rules. However this is not as simple with PF on FreeBSD. You cannot provide a limit on the packet rate per rule. You can limit the connection rate (see max-src-conn-rate), but I’m not sure this could be of any use here. It should be possible to use altq but this is not part of the GENERIC kernel. I really didn’t want to compile a custom kernel just as a workaround.

Instead I used another DHCPv6 client, namely ISC DHCP client (isc-dhcp43-client). Just create /usr/local/etc/dhclient6.conf and configure your DUID:

interface "igb0" {
  send dhcp6.client-id <DUID>;
}

On FreeBSD, isc-dhcp43-client doesn’t come with any rc starting script, so here is one for DHCPv6 (you should place it in /usr/local/etc/rc.d/dhclient6:

#!/bin/sh
#
# PROVIDE: dhclient6
# REQUIRE: DAEMON
# KEYWORD: dhcp
#
# Add the following lines to /etc/rc.conf to enable dhclient6:
#
# dhclient6_enable="YES"
#

. /etc/rc.subr

name="dhclient6"
desc="ISC DHCPv6 client"
rcvar="dhclient6_enable"

start_cmd="dhclient6_start"
stop_cmd="dhclient6_stop"

dhclient6_start()
{
  /usr/local/sbin/dhclient -cf "${dhclient6_conf}" -P -v "${dhclient6_iface}"
}

dhclient6_stop()
{
  if [ -r "${dhclient6_pid}" ]
  then
    kill -- -$(cat "${dhclient6_pid}")
    rm -f "${dhclient6_pid}"
  fi
}

load_rc_config ${name}

: ${dhclient6_enable="NO"}
: ${dhclient6_pid="/var/run/dhclient6.pid"}
: ${dhclient6_conf="/usr/local/etc/dhclient6.conf"}
: ${dhclient6_iface=""}

run_rc_command "$1"

Finally enable this in /etc/rc.conf:

dhclient6_iface="igb0"
dhclient6_enable="YES"