Source based routing

My two home servers are down for the moment. This also means that our two IPv6 SixXS tunnels are down which costs us 100 ISK per week. Argh! I need to get these up and running as soon as possible. Fortunately we have another VPS on Linux that can save us. So we just have to enable the two tunnels there and make sure that we can ping to/from both interfaces.

Setting up the two tunnels is easy. Use one configuration file per tunnel. Ensure that you change the parameters “tunnel_id” to the tunnel associated to this configuration file, one “pidfile” and “ipv6_interface” for each tunnel and “defaultroute” to false because we already have a default IPv6 route. Now you can start/stop each tunnel with:

aiccu start /etc/aiccu/tunnel0.conf
aiccu start /etc/aiccu/tunnel1.conf

aiccu stop  /etc/aiccu/tunnel0.conf
aiccu stop  /etc/aiccu/tunnel1.conf

Don’t forget to hack /etc/init.d/aiccu to start/stop both tunnel on each reboot. OK! So now ifconfig list the two interfaces, up and running sixxs0 and sixxs1. This is great but wait… Nobody outside can ping these interfaces. The tunnel must ping to be considered active by SixXS so we better get this running.

For now we have these three interfaces and IPs (not the actual names/IPs):

  1. net0 (2001::1) default
  2. sixxs0 (2a02::1)
  3. sixxs1 (2a02::2)

By default, all our IPv6 traffic goes through net0. However and unsurprisingly our ISP filters the traffic at the output of net0. So we cannot use this interface to answer the echo-requests. Actually, what we want is that traffic originating 2a02::1 goes through sixxs0 and from 2a02::2 goes through sixxs1. That is, one default route based on the source address.

Linux has long had support for multiple routing tables (CONFIG_IP_MULTIPLE_TABLES). Basically what we will do here:

  • Create two routing tables for each tunnel interface (sixxs0, sixxs1).
  • Each table will have a default route through its interface.
  • Lookup into one of the two tables according to the source IP.

You can find some relevant documentation in Linux Advanced Routing & Traffic Control, Chapter 4.

We first list the actual rules:

# ip rule list
0:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

We can see that we have three routing tables. One for the local addresses, the normal routing table (what you get with ip -6 route) and the fallback default table.
Let’s first check the local routing table (we are just curious):

# ip -6 route list table local
local ::1 via :: dev lo  proto none  metric 0  mtu 16436 advmss 16376 hoplimit 0
local 2a02::1 via :: dev lo  proto none  metric 0  mtu 16436 advmss 16376 hoplimit 0
local 2001::1 via :: dev lo  proto none  metric 0  mtu 16436 rtt 6ms rttvar 7ms cwnd 10 advmss 16376 hoplimit 0
local 2a02::2 via :: dev lo  proto none  metric 0  mtu 16436 advmss 16376 hoplimit 0
local fe80::1 via :: dev lo  proto none  metric 0  mtu 16436 advmss 16376 hoplimit 0
local fe80::2 via :: dev lo  proto none  metric 0  mtu 16436 advmss 16376 hoplimit 0
ff00::/8 dev net0  metric 256  mtu 1500 advmss 1440 hoplimit 0
ff00::/8 dev sixxs1  metric 256  mtu 1280 advmss 1220 hoplimit 0
ff00::/8 dev sixxs0  metric 256  mtu 1280 advmss 1220 hoplimit 0

So now you know what’s going on when you ping one of your local interfaces. But back to our point. We name our two new routing tables in /etc/iproute2/rt_tables:

# SixXS tables
200 sixxs0
201 sixxs1

Now we add the default route in each of these two tables:

ip -6 route add default dev sixxs0 table sixxs0
ip -6 route add default dev sixxs1 table sixxs1

And finally we use two rules to map the source address to the correct routing table:

# ip -6 rule add from 2a02::1 table sixxs0
# ip -6 rule add from 2a02::2 table sixxs1
# ip -6 rule list
0:  from all lookup local
16383:  from 2a02::1 lookup sixxs0
16383:  from 2a02::2 lookup sixxs1
32766:  from all lookup main
32767:  from all lookup default

It should be OK but let’s check that. We can ping from the sixxs interfaces:

ping6 -c1 -I 2a02::1 www.kame.net
ping6 -c1 -I 2a02::2 www.kame.net

We also check that we can ping our interfaces from another host:

ping6 -c1 2a02::1
ping6 -c1 2a02::2

Everything works, that’s great! Finally we just hack /etc/init.d/aiccu to configure the routing tables on each reboot. Note that you need to sleep a bit when you issue the aiccu start because the daemon needs a bit of time to enable the tunnels. Also note that you must be careful when you test your script (quoting the SixXS FAQ):

“If a client connects more than 4 times in 60 seconds (1 minute) the client will not be allowed to connect again for the next 5 minutes. In case this threshold is exceeded more than once in 24 hours a client will be automatically blocked for a week.”

As you can guess, I have been blocked. Oops!

SANE USB permissions

Today I had a permission problem with SANE on Linux. SANE stands for “Scanner Access Now Easy”, it provides standardized access to scanner hardware (http://www.sane-project.org) and this is the most commonly used scanning tool on UNIX/Linux.

In my case the USB scanner was not recognized when issuing scanimage -L from my user account although it worked correctly under root and my user is in the scanner group. What more is sane-find-scanner reported permissions errors while running the command as user. The owner and group for the device (in my case it was /dev/bus/usb/002/004) were root:root. At this point we already know that something weird is happening and I expected something like root:scanner instead.

Looking into /lib/udev/rules.d/60-libsane.rules, the line in charge of changing the permissions for each scanner device matched by SANE:

ENV{libsane_matched}=="yes", RUN+="/bin/setfacl -m g:scanner:rw $env{DEVNAME}"

This is nice but I do not use ACL and they are disabled in kernel,  so this command is useless. So I replaced this line with:

ENV{libsane_matched}=="yes", RUN+="/bin/setfacl -m g:scanner:rw $env{DEVNAME}", MODE="0664", GROUP="scanner"

Now the owner and group are correctly set to root:scanner and I can use my scanner as a regular user.

Note that on my system the libsane, sane-utils and xsane are the only packages depending on the acl package. According to what I’ve seen in the ChangeLog they do so in order to cope with MFP which I presume should be accessible as a scanner and printer device at the same time. What I would have done instead would be to create special group for MFP devices and use this instead. IMO still less of a mess than enabling ACL on the whole system for a single package.

GTalk browser plugin on Debian (testing)

So you installed the GTalk browser plugin on Debian testing and it doesn’t work. However GTalk is listed correctly when you list the plugins in your browser. So what now?

Well you can try to remove libudev0. It seems that the plugin has some problems when both libudev1 and libudev0 are present on the system.

Sound problem with zsnes on Debian amd64

You may have some sound problems with zsnes on Debian amd64 (jessie) especially if you want to use pulseaudio with SDL. There are a lot of old posts about these kind of sound problems with zsnes though none of the proposed solutions will work anymore. The last zsnes version (1.510+bz2-5) comes with only the SDL audio output. I configured the SDL audiodriver to pulseaudio but it had to be configured to pulse for zsnes to work (well don’t know why). Anyway all I had to do was this :

export SDL_AUDIODRIVER=pulse
zsnes

And now it works. So you may just add an alias or wrap this into a script to do that automatically.

Conntrack table flood

Recently we had problems with our gateway, connections were dropped and so on.
After a bit of investigation we found that it was due to a bugged game using Javascript which, when it ran on Firefox, opened connections in a loop flooding the connection tracking table in a matter of hours. Once found, it was easy to fix. This was also the occasion to tighten the timeouts values of nf_conntrack a little bit. Indeed 5 days timeouts for established connection doesn’t really make sense when your public IPv4 change every 36hours or so.

Start XScreensaver before going to sleep

It seems rational to request XScreensaver to lock your screen when you suspend your machine. This is possible in Debian via the /etc/default/acpi-support file, especially with this line :

# Comment this out to disable screen locking on resume
LOCK_SCREEN=true

However for some obscure reason this will lock the screen (i.e. issue the “xscreensaver-command -lock” command) after suspend, that is on resume. Since the locking process is not immediate your desktop will be available for anyone to watch (and use) for a duration of about one or two second. There is no need to say that this is unacceptable.

It is possible to avoid that by disabling the default screen locking mechanism and hooking it manually to PM. So you should add a script into /etc/pm/sleep.d. The following script is the first version of the script I used (beware it doesn’t work, see below) :

#!/bin/sh
# XScreensaver should be called BEFORE going to sleep to avoid the desktop
# to be shown for a few seconds when the system resumes from sleep.

case "$1" in
  hibernate|suspend)
    xscreensaver-command -lock
    sleep 1 # annoying sleep
    ;;
  *)
    exit 0;;
esac

You may notice that the script issues a sleep just after the xscreensaver-command has returned. It ensures that the screen will be really locked when the system effectively enters into sleep. This is needed because the xscreensaver-command will not lock the screen immediately, that is it is non-blocking in a certain way and you cannot ensure that the screen is effectively locked as soon as the command has returned.

However the script above doesn’t work. As Marcus Moeller commented, the above script won’t work by default on Debian and probably with most other distributions. That is because we don’t issue the xscreensaver lock command as the user owning the xscreensaver daemon. I quote his solution here :

#!/bin/sh
# XScreensaver should be called BEFORE going to sleep to avoid the desktop
# to be shown for a few seconds when the system resumes from sleep.

IS_ACTIVE="$( pidof /usr/bin/xscreensaver )"

case "$1" in 
  hibernate|suspend) 
    # check if xscreensaver is running. if not, just skip on. 
    if [ -z "$IS_ACTIVE" ] 
      then : 
      else 
      # run the lock command as the user who owns xscreensaver process, 
      # and not as root, which won't work.
      su "$( ps aux | grep xscreensaver | grep -v grep | grep $IS_ACTIVE | awk '{print $1}' )" 
             -c "/usr/bin/xscreensaver-command -lock" &
      sleep 1
    fi
    ;;
  *)
    exit 0;;
esac

Digging in xscreensaver’s code shows that what the command actually needs is a connection to the X server. If xscreensaver-command cannot find the display from either command line or environment variables, it will fall back to “:0.0“. But this will fail if root cannot connect to the X server (which is generally the case). That’s how the ‘user approach’ fixes it. However this won’t work anymore if there are multiple instance of xscreensaver running on different displays (only one of them will be locked). Another solution would be to issue the command on each display where root can connect to. However this poses two problems :

  1. It is not as easy as it seems to reliably list all available displays. (see http://stackoverflow.com/questions/11367354/obtaining-list-of-all-xorg-displays).
  2. It requires that each lockable session allows connections from root with “xhost si:localuser:root“.
Here is the modification I posted in response which uses the ‘display approach’ instead:
#!/bin/sh
# XScreensaver should be called BEFORE going to sleep to avoid the desktop to be shown
# for a few seconds when the system resumes from sleep.

case "$1" in
  hibernate|suspend)
  # The X server may not be running
  if [ ! -d /tmp/.X11-unix ]
  then
    exit 0
  fi 

  # Lock each available display
  for socket in $(ls /tmp/.X11-unix)
  do
    display=$(echo "$socket" | tr "X" ":")
    xscreensaver-command -display "$display" -lock
  done

  sleep 1 # annoying sleep
  ;;
  *)
   exit 0;;
esac

However we are not done yet. As you can see we still rely on sleep to ensure that the screen is locked before our script returns control to the suspend procedure. With usage it became clear that one second was not sufficient as the script would return too early from time to time. Incrementing the duration of the sleep would be more than annoying and it doesn’t offer any real guarantee anyway. The only solution would be to find a way to exit the script when we are sure that the display is effectively locked. This is possible by watching at the changes of states of the screensaver while issuing the lock command. There is a slight last problem however. If multiple displays are present we want to issue that “lock ‘n watch” procedure in paralell to avoid accumulating the locking delays. That’s the solution I use in the script below, note that we don’t rely on sleep anymore:

#!/bin/sh
# XScreensaver should be called BEFORE going to sleep to avoid the desktop to be
# shown for a few seconds when the system resumes from sleep.
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin

lock_display() (
  socket="$1"
  display=$(echo "$socket" | tr "X" ":")

  # Temporary pid file for the watching command
  tpid=$(mktemp)

  # Wait until the display is actually locked.
  (timeout 2s xscreensaver-command -display "$display" -watch & echo $! > $tpid) | (
    # Issue the lock command only when we know that
    # the watching pipe is ready.
    xscreensaver-command -display "$display" -lock

    while read line
    do
      line=$(echo $line | cut -d' ' -f 1)

      if [ "$line" = LOCK ]
      then
        # We have to kill the watching command manually before breaking.
        kill -TERM $(cat $tpid)
        break
      fi
    done
  )

  rm $tpid
)

case "$1" in
  hibernate|suspend)
    # The X server may not be running
    if [ ! -d /tmp/.X11-unix ]
    then
      exit 0
    fi

    # Lock each available display
    for socket in $(ls /tmp/.X11-unix)
    do
        # Lock the display
        lock_display $socket &
    done

    # Wait until every displays are locked
    wait
    ;;
  *)
    exit 0;;
esac

As stated above you still need to allow connections from root to your display. You may for example use this command when your session start :

xhost si:localuser:root

Or, as the man page of xhost states, use the file /etc/X*.hosts to do that globally.

Default applications with GTK-3, Chromium and beyond

Today I was surprised to see a GTK-3 application opening an HTTP URL with Opera. I don’t use  Opera and I just installed by curiosity long ago and forgot about it.  I configured the Debian alternatives however GTK-3 seems to use xdg-mime as confirmed with an strace of the concerned application and references to /usr/share/applications/defaults.list. Note that you may have to create a symlink for defaults.list to /usr/share/applications/mimeapps.list.

You can use the xdg-mime command to configure the default application for each protocol:

$ xdg-mime default chromium.desktop x-scheme-handler/http
$ xdg-mime default chromium.desktop x-scheme-handler/https

You can also configure this manually by editing /usr/share/applications/defaults.list.  Just add these two lines:

x-scheme-handler/http=chromium.desktop
x-scheme-handler/https=chromium.desktop

In the [Default Application] section. Thunar and Chromium also use this so you can configure them to open PDF and handle Skype calls properly. See:

[Default Applications]
x-scheme-handler/http=chromium.desktop
x-scheme-handler/https=chromium.desktop
x-scheme-handler/skype=skype.desktop
text/html=chromium.desktop
application/pdf=mupdf.desktop

This should do the trick. You can get the list of available MIME types with this command (note that the path depends on the location of the shared-mime-info database):

find /usr/local/share/mime -name "*.xml" -exec cat {} \; |g -E -o "type=\".*\"" | sort | uniq

Otherwise you may also use the dconf-editor from the dconf-tools package or gconf-editor to configure GNOME-2/3 default applications. With gconf you should search the following keys:

/desktop/gnome/applications
/desktop/gnome/url-handlers

With dconf instead you should search for:

/desktop/gnome/url-handlers
/org/gnome/desktop/applications

Get rid of that Non-Breaking space

The non-breaking space is a variant of the space character which as the name suggests prevents automatic line breaking when using a space character. Another common use of it is to avoid collapsing of white-spaces in formats such as TeX or HTML. There are also some others specific typographic uses but nevermind.

From my point of view this character is a real pain, a annoying remain of a former era when the only editor you had was plain-text and which is now just good at invisibly polluting your work so that no compiler, interpreter or any other ASCII oriented tool would accept it anymore. For ASCII is not dead, it’s 7-bit, it’s common, it’s the neutral zone of all character set and No-Break Space shall not interfere therefore it must be eradicated from the very surface of earth or at least from my own keyboards.
There are several options to do that under Xorg. The first one would be to use the setxkbmap command to specify the nbsp:none option for your keyboard.
setxkbmap -option "nbsp:none"

However this won’t work as you might expect it especially when you have multiple keyboards with different layouts. So another solution would be to specify XkbOption directly within the Xorg configuration file, one for each input device. But this won’t work neither if you use a Bépo layout as this option will simply disable the use of underscore (AltGr Space). So the final solution is based on xmodmap to modify the list of keysyms assigned to the space keycode (0x41). You may just add the following line to ~/.xmodmap and ensure that the file is loaded when your session starts with xmodmap ~/.xmodmap.

keycode 65 = space space space space underscore underscore space space

Intel NIC connection problem

PCIe ASPM is an hardware power management protocol for PCI express devices. It allows a far better power management than what can be done with software power management at the price of an increased latency to the device. However some hardware don’t advertise it properly. And this was in fact the origin of the power regression case of 2.6.38 which was later fixed in 3.3.

It may also cause some Intel NIC (such as e1000e) to fall asleep unintentionally. So if you have an Intel NIC on Linux and it even fails to acquire an IPv4 address then you might try to add this option to your kernel command line : pcie_aspm=off. On Debian you can add this option to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub and issue an update-grub.

You may also want to change the link-state of one specific device to L0/L1 at runtime. For this I’ll point you there : http://wireless.kernel.org/en/users/Documentation/ASPM.

Raspberry Pi Ethernet speed

I’ve been a long time user of IPv6 tunnels from SixXS to provide an access to the IPv6 Internet behind my ISP. These tunnels also allow me to use static IP addresses for my home servers along with static AAAA records and this is cool !

Currently I use several Debian GNU/Linux based soft-routers with two (100 and 1000) Ethernet ports. These are often running on old recycled laptops which consumes around 40 Watts of power at peak level. Next to that the ARM Raspberry Pi platform consumes around 3 Watts of power (though I still have to measure it by myself). So I thought about replacing all my home-routers with those.

However the Raspberry Pi model B uses a SMSC LAN951x chip which includes the USB 2.0 Hub and an 10/100 Ethernet controller on top of it (which is known as smsc95xx in the Linux kernel). My  main concern was that it would not be fast enough to support the IPv6 tunnel at its peak bandwidth of 60Mbps (that is 30Mbps downstream/upstream).

I already use one RPi as an experimental home-router here. Our Internet bandwidth is a bit slow (12Mbps) so the USB-Ethernet  shouldn’t be a problem. I’ve conducted quick tests with IPerf and as you can see the results are pretty good as long as it doesn’t involve I/O on the RPi.

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.0.0.1 port 5001 connected with 10.0.0.3 port 37373
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.1 sec 114 MBytes 94.4 Mbits/sec
[ 5] local 10.0.0.1 port 5001 connected with 10.0.0.3 port 37376
[ 5] 0.0-10.1 sec 114 MBytes 94.5 Mbits/sec
[ 4] local 10.0.0.1 port 5001 connected with 10.0.0.3 port 37377
[ 4] 0.0-10.1 sec 114 MBytes 94.6 Mbits/sec
[ 5] local 10.0.0.1 port 5001 connected with 10.0.0.3 port 37378
[ 5] 0.0-10.1 sec 114 MBytes 94.5 Mbits/sec