Constant SD-Card corruption on the RPi

Our home servers broke. Here we are again.

I spent weeks of my time, countless evenings up to 4AM, entire weekends since months trying to design and configure our reborn home-servers and gateways.

And it was neat.

  • DNSSEC all the way down
  • RPC accross the nodes
  • Easy configuration
  • Caching and stuff
  • Automatic tests

It took me a lot of time to assemble all of this in something that I liked. And to document everything so that we could easily install a new node from scratch.

I installed two nodes and it worked well for several weeks. Until a week ago or so I started to see corruption on the first node. And by corruption I mean random garbage in a lot of binaries and libraries. Exec format error at every corner. At this point it was completely broken and useless so the only option was to reinstall it.

So I used a new SD-Card, changed the power supply and reinstalled everything last weekend. Just finished today and also fixed bugs in some of our scripts. Had to search for a package on the second node which at this point was still in a pretty good shape.

$ apt-cache
zsh: exec format error: apt-cache
$ su
zsh: exec format error: su

Dang! So there goes another weekend I will spend to reinstall the thing. And who knows how long until the first node gets corrupted again.

Checked the TP1-TP2 voltage, 4.65V, probably because of the second USB Ethernet adapter. I tried to limit the amount of writes on the SD-Card. No heavy writers, no swapping, no overclocking.

So I must be doing something wrong, right? Right?! The RaspberryPi can be that unreliable. I wonder how many power supplies and SD-Cards I will have to buy and try until, by sheer luck, I do not have to reinstall everything in the following three months or so.

I ran into this problem years ago. And now it seems that I will run in the same problem over and over again. Any recommendation is welcome of course. Though to be honest, for now, I just want to fly the damn thing across the room.

Switch MTA on FreeBSD

As you probably know FreeBSD comes with Sendmail installed as the default MTA. However this may be a bit overkill on a desktop installation where the most you might want is to relay mails to an external address. Luckily it is quite easy to change the default MTA as described in the handbook, see 28.4. Changing the Mail Transfer Agent.

On my Desktop I prefer to install nullmailer. This is a simple MTA replacement for hosts which only relay mails through a smart relay. GNUTLS (SSL) is not enabled by default in the nullmailer package on FreeBSD. So if you want SSL you have to compile the port. This is my case. Let’s install it:

cd /usr/ports/mail/nullmailer
make install clean
(...)
pkg lock nullmailer

The configuration happens in /usr/local/etc/nullmailer. This directory contains multiple files and each one of them focuses on a specific aspect of the configuration.

First we specify the remote SMTP through which our mail shall be relayed, this is the remotes file. This file contains a list of remote servers, the module used to send the message and command-line arguments for that module. Modules are located in /usr/local/libexec/nullmailer. The man page states that you can list available options using --help on each protocol module.

In most cases you want to use the smtp module which takes the following arguments (with SSL enabled):

  • port: SMTP port (25, 465, 587, …)
  • user: SMTP user
  • pass: SMTP password
  • auth-login: LOGIN authentication method (default to PLAIN)
  • ssl: Use SSL/TLS encryption
  • starttls: Use STARTTLS command to initiate encrypted connection
  • insecure: Accept invalid certificates (which I do not recommend)
  • x509certfile: Client certificate file
  • x509cafile: Certificate Authority trust file (default to /etc/ssl/cert.pem on FreeBSD)
  • x509crlfile: Certificate revocation list
  • x509fmtder: Switch from PEM to DER format for the certificates

Here is an example that would relay through relay.example.com:465 using SSL and LOGIN authentication:

relay.example.com smtp --port=465 --ssl --auth-login --user=some-user --pass=some-password

Since this file contains your SMTP password in cleartext, I advise you to:

chown nullmail:nullmail remotes
chmod 600 remotes

Next we edit the name that will be used to construct email addresses on this host. You configure this in the me file. Normally this should be the fully-qualified host name of the computer running nullmailer. This is really useful to distinguish, say root at machine-a from root at machine-b. However some mail providers refuse to relay mails from a different domain name than their own so it might be useful to change this in those cases (I am my own mail provider, so personally I don’t care and do what I want). You also need to configure defaultdomain to your domain name. That is your FQHN minus the hostname. If a mail is sent to an address that is not localhost and does not contain a domain name (no period in the hostname), this domainname will be appended to it.

After that we configure the mail to which all local mails are forwarded. You configure this address in the adminaddr file. And we also configure the file pausetime. This is the interval of time between two queue runs with a default value of 60 seconds. I prefer to set this to a higher value, like 15 minutes.

For more information about the configuration of nullmailer, see this article. Although related to Raspbian on a RPi, it remains mostly the same.

Now we need to replace the MTA on FreeBSD. First we configure the mailwrapper (see man mailwrapper) in /etc/mail/mailer.conf. Replace each line with their nullmailer equivalent, that is:

sendmail  /usr/local/libexec/nullmailer/sendmail
send-mail /usr/local/libexec/nullmailer/sendmail
mailq     /usr/local/libexec/nullmailer/mailq

Time to test. Disable sendmail, enable nullmailer and send a mail. Oh and by the way, tail -f /var/log/maillog in any case:

service sendmail stop
service nullmailer onestart
echo Hello from FreeBSD\! | mailx -s "test" root

If it works, you can now disable sendmail and enable nullmailer in /etc/rc.conf:

sendmail_enable="NONE"
nullmailer_enable="YES"