Custom user Xsession with SDDM

Debian stretch is out, a lot of obsolete packages, a lot of major upgrades, which all in all resulted in quite a painful transition the last few days. But I’ll tell you more about that in the following posts.

I don’t really spend much time on Linux nowadays so KDE (along with KDM) has always been my goto solution for a jack all trade no-BS works-out-of-the-box desktop environment. And it worked like that just fine, until… well you know how software goes. KDE has been upgraded, KDM has been depreciated and replaced with SDDM.

I also use xsession so that I have a common way of starting session scripts and daemons (such as this one) and configuring stuff across different desktops. I generally selected custom session in the display manager and that was it. But SDDM does not seem to provide a way to do so, or at least that’s not so clear.

By default, it will execute /etc/sddm/xsession which itself sources /etc/X11/Xsession to which it will pass as argument the value of the Exec line in the desktop file (located in /usr/share/xsessions) describing the currently selected session.

If we want to bypass this, we need to scrap the argument passed to /etc/X11/Xsession no matter what SDDM thinks the current session should be. To do so create a wrapper for Xsession in /etc/X11/user-Xsession:

#!/bin/sh

export PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin

# Discard argument, we don't care about selecting the desktop environment.
/etc/X11/Xsession

And now configure SDDM to use this instead of its own version of it, in /etc/sddm.conf:

[X11]
SessionCommand=/etc/X11/user-Xsession

Using NAT64 locally

NAT64 is a translation mechanism that allows IPv6 addresses to be transparently mapped to IPv4 addresses. The principle if very simple. In a NAT64 address, the IPv4 address is embedded into the 32 least significant bits of the IPv6 address. When you send an IPv6 packet to a NAT64 router, it will extract the IPv4 address embedded into the NAT64 IPv6 destination address and send the packet using this IPv4 instead. The source IPv4 address used is chosen within a provided pool of IPv4 addresses (which can be SNATed, in which case you can use a private network reserverd for the NAT64 pool). The NAT64 router will also do the reverse mapping and translate the reply from the IPv4 remote address to your original IPv6 address.

This is very convenient for IPv6 only hosts (such as IoT smart objects) that need to contact a very sparse IPv6 Internet. Even if a remote host is only accessible via IPv4, it can still be accessed from IPv6 using its NAT64 address. Also since the majority of addresses are resolved using DNS, there are some servers (unbound is one of them) that provide a DNS64 module which will automatically construct a NAT64 AAAA record when no other IPv6 records are found for a specific domain. This way your IPv6 only hosts using DNS believe that all domains have an AAAA record, and that all the Internet is IPv6 enabled.

In my case I use Tayga NAT64 router along with Unbound DNS64 module. There are several posts out there that explain how to configure these two and this is not my intent here. Instead I want to focus on a problem that I had using Tayga’s NAT64 locally. That is trying to ping a non-IPv6 domain from the host that is running Tayga:

$ ping6 ipv4.google.com

This command worked fine from outside the host running Tayga (for packets routed through the NAT64 router), but didn’t work at all when issued on the host running Tayga.

Using tcpdump, I found that when issued locally, Tayga used its own IPv4 address (its ipv4-addr option) as source address for the translated IPv6 packets. Since this address was not mapped in Tayga IPv4 source address pool, it did not translate the reply back to the original IPv6 address and so the ping6 command never got any reply.

What I also found was that the source address used by the ping6 command was also Tayga’s own IPv6 address (its ipv6-address option). Since the NAT64 address that ping6 tries to use is routed through the nat64 interface, it defaulted to the nat64 interface IPv6 address (that is Tayga’s IPv6 address) for its source address.

This is what triggered Tayga to use its own IPv4 address which is outside its pool resulting in the reply not being translated back to its own IPv6 address. So we need a way to avoid choosing Tayga’s own IPv6 address when trying to reach a NAT64 address locally. We know that Tayga configures its own IPv4 and IPv6 addresses on its nat64 interface. All we have to do is to add another IPv6 address to the interface so that it will be used instead as the source address for local packets destinated to the nat64 router interface.

Suppose we have:

  • Internet routed prefix: 2001:db8:1::/64
  • NAT64 IPv6 prefix: 2001:db8:64::/96
  • NAT64 IPv6 address: 2001:db8:1::64
  • NAT64 IPv4 address: 192.168.1.64

Then the nat64 interface will have the 2001:db8:1::64 IPv6 address. Just add another IPv6 address to this interface with:

ifconfig nat64 inet6 add 2001:db8:1::46 netmask 128

Tadaaa! Now when you try to reach a NAT64 address locally, the address you just added to nat64 will be used instead as the source address. Since this is not Tayga’s own IPv6 address, it will dutifully map the NAT64 using its IPv4 address pool and translate the reply back to the new IPv6 address. In other words NAT64 addresses now work on the host running the NAT64 router.

Linuxulator ist kaputt

Since a few weeks I’m running RELEASE on a custom kernel to use a patch that I made for a missing feature in the IPv6 stack (namely icmp_may_rst).

But a few minutes ago I had the surprise to find that the Linuxulator was no longer working. Trying to run a Linux binary failed with the following error:

ELF binary type "0" not known.
exec: test: Exec format error

Actually looking at kldstat, the Linux kernel module wasn’t even loaded. Trying to load it manually gave me the following error:

link_elf_obj: symbol kern_sched_setscheduler undefined
linker_load_file: Unsupported file type

OK so what is this sched_setscheduler you are talking about? Well there you go!
Now the Linuxulator depends on that syscall, but for some reason the necessary option disappeared from my custom kernel configuration. All I had to do was to add it again and recompile:

options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions

And now I can run Linux binaries again!

Redirect traffic to loopback

Today I wanted to transparantly redirect the DNS requests coming at the output of a tunnel to a local caching DNS resolver. The caching DNS was listening only on the loopback as port 53 was already bound to other interfaces. That would be fairly simple on Linux:

echo 1 > /proc/sys/net/ipv4/ip_forward

iptables -t nat -A PREROUTING -i tun0 -p udp --dport 53 -j DNAT --to-destination 127.0.0.1
iptables -A FORWARD -i tun0 -o lo -p udp --dport 53 -j ACCEPT

But… The kernel will refuse to route packets with the loopback as source or destination because this qualify as a martian packet. The solution was to enable the route_localnet flag. As stated in the kernel documentation:

route_localnet – BOOLEAN: Do not consider loopback addresses as martian source or destination while routing. This enables the use of 127/8 for local routing purposes (default FALSE).

This is per interface. So I just had to enable this on the tunnel interface:

echo 1 > /proc/sys/net/ipv4/conf/tun0/route_localnet

Rename interfaces on Linux

I just reinstalled a Debian stable on a laptop but messed with the interfaces so that an external USB WiFi card appeared as wlan0 while the main card appeared as wlan1. In case you wondered you can rename or reset interface names in /etc/udev/rules.d/70-persistent-net.rules. That’s on systemd though.
I wonder how we can change that on sysvinit? Nobody cares, probably, but I do.

According to what I read there, it is not consistent. Interfaces are named in the order in which they appear during the boot process. However it is possible to use ifrename from the wireless tools package. Why this tool that should work for all type of interface is part of the wireless tools package is beyond my comprehension. But hey whatever, Linux, and it just works.

If you are curious and want to know how ifrename actually does rename an interface, according to the code it uses a SIOCSIFNAME ioctl on a socket file descriptor. There it passes a struct ifreq in which you can provide a new name for the interface. Just man netdevice(7) for more info.

Constant SD-Card corruption on the RPi

Our home servers broke. Here we are again.

I spent weeks of my time, countless evenings up to 4AM, entire weekends since months trying to design and configure our reborn home-servers and gateways.

And it was neat.

  • DNSSEC all the way down
  • RPC accross the nodes
  • Easy configuration
  • Caching and stuff
  • Automatic tests

It took me a lot of time to assemble all of this in something that I liked. And to document everything so that we could easily install a new node from scratch.

I installed two nodes and it worked well for several weeks. Until a week ago or so I started to see corruption on the first node. And by corruption I mean random garbage in a lot of binaries and libraries. Exec format error at every corner. At this point it was completely broken and useless so the only option was to reinstall it.

So I used a new SD-Card, changed the power supply and reinstalled everything last weekend. Just finished today and also fixed bugs in some of our scripts. Had to search for a package on the second node which at this point was still in a pretty good shape.

$ apt-cache
zsh: exec format error: apt-cache
$ su
zsh: exec format error: su

Dang! So there goes another weekend I will spend to reinstall the thing. And who knows how long until the first node gets corrupted again.

Checked the TP1-TP2 voltage, 4.65V, probably because of the second USB Ethernet adapter. I tried to limit the amount of writes on the SD-Card. No heavy writers, no swapping, no overclocking.

So I must be doing something wrong, right? Right?! The RaspberryPi can be that unreliable. I wonder how many power supplies and SD-Cards I will have to buy and try until, by sheer luck, I do not have to reinstall everything in the following three months or so.

I ran into this problem years ago. And now it seems that I will run in the same problem over and over again. Any recommendation is welcome of course. Though to be honest, for now, I just want to fly the damn thing across the room.

Where did my PGP keys go?

Today I noticed that one of my PGP private key just disappeared of GPG. The key did not appear when I did gpg --list-secret-keys. After a bit of investigation I discovered that the problem did not affect Linux hosts but only FreeBSD hosts. Weird…

The source of the problem was a migration from GnuPG v2.0 to v2.1. According to this page, GPG does not handle the private keys anymore and delegates all private keys operations to the gpg-agent. Therefore GPG v2.1 migrates the legacy secret keyring, secring.gpg, to the gpg-agent key store, private-keys-v1.d and then forgets about it.

Though, you see, my GPG keyrings were synchronized across all hosts. But the GnuPG package on Debian is still v2.0, while FreeBSD is v2.1. Get the picture?

I synced my keyring on FreeBSD hosts where GPG migrated my private keys to the gpg-agent key store. Then I generated a new key pair on a Debian host, which was added to the legacy keyring. Resynced, but the newer version of GPG didn’t care, they already migrated to the new key store.

Fortunately it was easy to fix, all you have to do is re-import your legacy keyring with one of the newer versions of GPG. The private keys are now also present in the new key store so you can sync to all other hosts.

gpg --import $HOME/.gnupg/secring.gpg
gpg --list-secret-keys

IPs ban on Linux

Ban Hammer

Who needs a quick ban?

Today we had a bruteforce attack on our nginx server. Well cannot say he was anywhere near successful though, the guy did POST /wp-login.php several times per second and all he got as an answer was 404. Fat chance…

But still, he had our access logs growing far larger than they usually do. So I tried to ban him. Unfortunately nginx does not use TCP wrappers by default (you can use ngx_tcpwrappers although it will have to be rebuilt from source).

So I made a little script, called ban-hammer to temporarily ban IPs using IPTables. There is also a cron.daily script to unban IPs each day. The script requires rpnc, but it is easy to adapt without it.

These scripts add and remove the IPs into a special IPT chain (which you can configure in the script). So you also have to configure your firewall to jump to the two chains and load banned IPs on boot:

echo "Bans"

load_bans() {
  ban_table=$1
  ban_chain=$2
  iptables=$3

  $iptables -N $ban_chain

  while read ban
  do
    ip=$(echo "$ban" | cut -d'=' -f 1)
    $iptables -A $ban_chain -s "$ip" -j DROP
  done < "$ban_table"

  $iptables -A INPUT -j $ban_chain
}

load_bans /etc/firewall/ip4.ban IP4BAN iptables
load_bans /etc/firewall/ip6.ban IP6BAN ip6tables

Hello FreeBSD!

It’s been more than two weeks now that I switched from Linux to FreeBSD. There are multiple reasons behind this change and I will not dwell on all of them. If you read this blog (do you? :)), you probably know that I am a long time advocate of Debian. One particular thing that I like with Debian is that it doesn’t tie your hand with a large set of packages. It is an universal operating system that you can tailor to better suit your needs.

However, as time passes it became harder to modify anything. More and more I find myself patching programs that just want to do things on their own fancy way. More and more some random daemon just gets in my way because it supposedly covers all possible use cases. And recently I came under the impression that my system was just a bunch of layers of layers of various daemons doing their stuff somehow, somewhere, all of them trying to reinvent the wheel, with a twist.

Finally there is one important thing you should remember, Linux is not UNIX. Actually in the past few years, it started to diverge from this philosophy quite significantly. This article presents some differences between the UNIX and the Linux/FLOS model much better than I could do. And this is where we come to the root of my decision. While I can understand some of the benefits of the later approach, it dawns on me that as an user, I do not fit in FLOS and if I keep using Linux as a desktop, this life will be a hell of frustration and ranting without end. Note that this transition was long time foreseeable. I always spent a lot of time with BSDs. However these were casual and experimental setups and I didn’t do much more than porting stuff to it.

I could as well use this system on a daily basis. So I decided to take the leap and use FreeBSD on my laptop (ThinkPad X201). I first installed FreeBSD 10 (RELEASE), but it didn’t work as expected. In particular the Intel KMS driver did not work properly. Also xrandr did not work, and the performances were far lower than Linux. Needless to say, I was a bit downhearted. I expected so much from this first installation.

After an evening weighting the pros and cons, sadly contemplating the idea of returning to Linux, I decided to give it another try with FreeBSD 11 (CURRENT). Fortunately almost everything worked perfectly then. The Intel KMS driver works, although I don’t have access to the ttys (ttys and suspend now work on HEAD). Xrandr works perfectly which is imperative to give a presentation. The wireless card, sound card, fingerprint reader and ultra base also work with no apparent problem.

However I still have some problems with the function keys not detected on the external ThinkPad keyboard. Also xscreensaver does not always detect the finger print reader. Finally the secondary mouse and keyboard are not always properly detected by X. I guess this is probably a problem with HAL. But I did not look into it yet (ums_load="YES" in /boot/loader.conf).

I did several quick benchmarks to compare the performances with the Debian installation. I will post the results in a few days. I will also leverage the change to update some of my projects and also to clean my configurations a little bit. I already did so for Emacs and Awesome WM, though for now I’ve something else to do.