Our home servers broke. Here we are again.
I spent weeks of my time, countless evenings up to 4AM, entire weekends since months trying to design and configure our reborn home-servers and gateways.
And it was neat.
- DNSSEC all the way down
- RPC accross the nodes
- Easy configuration
- Caching and stuff
- Automatic tests
- …
It took me a lot of time to assemble all of this in something that I liked. And to document everything so that we could easily install a new node from scratch.
I installed two nodes and it worked well for several weeks. Until a week ago or so I started to see corruption on the first node. And by corruption I mean random garbage in a lot of binaries and libraries. Exec format error at every corner. At this point it was completely broken and useless so the only option was to reinstall it.
So I used a new SD-Card, changed the power supply and reinstalled everything last weekend. Just finished today and also fixed bugs in some of our scripts. Had to search for a package on the second node which at this point was still in a pretty good shape.
$ apt-cache
zsh: exec format error: apt-cache
$ su
zsh: exec format error: su
Dang! So there goes another weekend I will spend to reinstall the thing. And who knows how long until the first node gets corrupted again.
Checked the TP1-TP2 voltage, 4.65V, probably because of the second USB Ethernet adapter. I tried to limit the amount of writes on the SD-Card. No heavy writers, no swapping, no overclocking.
So I must be doing something wrong, right? Right?! The RaspberryPi can be that unreliable. I wonder how many power supplies and SD-Cards I will have to buy and try until, by sheer luck, I do not have to reinstall everything in the following three months or so.
I ran into this problem years ago. And now it seems that I will run in the same problem over and over again. Any recommendation is welcome of course. Though to be honest, for now, I just want to fly the damn thing across the room.