Don’t forget the pipe subshell

This is a common error while using pipe over while loops. Consider this shell snippet:

#!/bin/sh

cat file.txt | while read line
do
  echo "inside loop"
  exit 1
done

echo "outside loop"
exit 0

You’d expect the script to exit on the first line in file.txt. However execute this script and you have:

inside loop
outside loop

It is as if the exit 1 inside the loop is ignored. Another example:

#!/bin/sh

a=0
cat file.txt | while read line
do
  echo "inside loop"
  a=1
done

echo "outside loop"
echo "a=$a"

Here you’d expect the value of a to be 1 at the end of the script. Instead, if you execute this you have:

inside loop
outside loop
a=0

It’s as if the variable a isn’t even updated. In fact it is, though only inside the loop. So what is happening here?

The pipe (|) you use to feed the loop creates a subshell. In fact this is really just another process. So the exit 1 or a=1 only apply to these piped processes.

How can you fix that?
In the simple case presented above, you can simply use file redirection:

while read line
do
  ...
done < file.txt

But what if you really want to feed the loop with the output of another process. Like you would do with find for instance.

If you use bash you can use process substitution as described here. But you shouldn't use bash for scripting anyway. For shell scripting you might be tempted to use a temporary file to store the process output:

# Use a temporary file.
tmp=$(mktemp)
find . > $tmp
while read line
do
  ...
done < $tmp 
rm $tmp

However this consumes disk space, and the loop only starts after the find process exited. Another option would be to use a named fifo:

fifo=$(mktemp -u)
mkfifo $fifo
find . > $fifo &

while read file
do
  ...
done < $fifo
rm $fifo

This time you create a single file, yet no disk space is used. Also the find command is a child process, so the loop reads find output as it comes.

Although the version above already works as it should, you may want to use an anonymous fifo. This way you only need to create a fifo file, although you can delete it immediatly. You can achieve this with a little help from our beloved

file descriptor 3 ❤

fifo=$(mktemp -u)

# Create fifo
mkfifo $fifo

# Create fd 3 and unlink fifo file.
exec 3<> $fifo
rm $fifo

# Redirect find to fd 3.
find . >&3 &

# Feed fd 3 to while loop.
while read line
do
  ...
done <&3

# Close fd 3.
exec 3>&-

Epson 3490 Scanner

This is an information that tends to be forgotten on the Internet, so I’m publishing it here. How to get an Epson Perfection 3490 Photo scanner running under Linux, FreeBSD or whatever. Paths may change on your system, so you may need to adapt the instructions below.

1. Install xsane.

2. Download the Epson firmwares. For the Epson 3490, you need esfw52.bin. You may find this file on Internet, although it tends to disappear. But in any case you can also find this file here.

3. Uncompress the firmwares. That is, sudo tar -Jxvf epson-firmwares.tar.xz -C /usr/local/share/sane.

4. Modify /usr/local/etc/sane.d/snapscan.conf, change the firmware line to point to the esfw52.bin firmware. That is following the commands above, change the firmware line to firmware /usr/local/share/sane/epson-firmwares/esfw52.bin.

If you are running FreeBSD

You should still ensure that you can use the scanner as a normal user.

5.Let’s change the owner of the scanner so that it’s available to users in the saned group. Create /etc/devd/saned.conf and add:

notify 100 {
  match "system" "USB";
  match "subsystem" "INTERFACE";
  match "type" "ATTACH";
  match "cdev" "ugen[0-9].[0-9]";
  match "vendor" "0x04b8";
  match "product" "0x0122";
  action "chown -L cups:saned /dev/$cdev && chmod -L 660 /dev/$cdev";
};

Notice the 0x4b8:0x0122, identifying the scanner USB device which you can get from the lsusb command while the scanner is plugged in.

6.Restart devd with service devd restart.

7.Add yourself to the saned group with sudo pw groupmod saned -m

8.You may need to log in again so that new group changes are taken into account.

Fastd and truncated IP

You might have guessed from the last post that I’ve been playing a bit with Fastd the last few days.

However I had some surprises after successfully configuring my first tunnel. Light traffic such as ping would go through without problem. But larger traffic such as rsync transfers or downloading/uploading files would hang endlessly.

At the remote side of the tunnel, I looked with tcpdump at the traffic out of the tun interface. Packets were going through, although packets with a large payload were apparently truncated:

truncated-ip6 - 4 bytes missing
truncated-ip - 4 bytes missing

Indeed large packets were 4 bytes shorter than intended both for IPv4 and IPv6 traffic. This sounded a lot like some kind of MTU mismatch in the tunnel itself. Still, both sides of the tunnel were configured with the same MTU. Also, Fastd ensures that both sides use the same MTU during the tunnel creation.

So what is it? I don’t know. I did not investigate further. But I’ve found a workaround.
Once the tunnel is created, I simply reduce the MTU on the entry/client side interface of the tunnel of 4B. That is in the establish script:

mtu=$(("$INTERFACE_MTU" - 4))
ifconfig "$INTERFACE" mtu "$mtu"

That’s still weird though. So I’d like to find some definitive answer as to why this is happening.

Fastd on FreeBSD

Fastd is nice and small secure tunneling daemon. A bit like OpenVPN, if you wish, but geared toward small devices, simpler in its design and in some ways more generic.

There was a FreeBSD port, but it has been marked as broken. The fix, however, is very simple, if you accept to get rid of AES128 and instead use the SALSA stream cipher:

cmake -DWITH_CIPHER_AES128_CTR=FALSE CMakeLists.txt
make
make install

Chromium on FreeBSD

Good news everyone! Chromium is now perfectly usable on FreeBSD.
The longstanding hanging tab bug has been resolved. See also PR 212812 and this this FreeBSD forum post.

This was fixed in r337328 but is not yet available in 11.2-RELEASE. Fortunately there are temporary fixes too that you can use while waiting for the patch to be included in the next release.

First add this line to /etc/sysctl.conf:

net.local.stream.recvspace=16384

Second use a memory backed filesystem for the chromium cache. A script to do so was included in the chromium package, but it has since been removed now that a proper fix is coming in.

But if you want to do this manually, first ensure that ~/.cache/chromium directory exists and is empty. Then in /etc/fstab add this line with $USER changed accordingly:

md /home/$USER/.cache/chromium mfs rw,late,noatime,noexec,nosuid,-w$USER:$USER,-s300m 2 0

This will mount the chromium cache path on an UFS partition over a memory backed virtual disk.

I’ve been testing this for several days now and it works like a charm. Don’t forget to remove this workaround when you are past r337328 though.

Random wallpapers

If you ever want to randomly change your wallpapers every few minutes, hours or whatever, I just made a script to do just that. You can find it here.

This script will at regular interval select a random file within the specified directory and use the specified command to use it as a wallpaper. For example if you want to change your wallpaper every 10 minutes with a picture in ~/Pictures/wallpapers.

wp-random.sh 600 ~/Pictures/wallpapers/ feh --bg-scale

Note that we use feh here to setup the wallpaper, but you can use any command you like.

You can use SIGUSR1 to redraw the current wallpaper (for example if you just enabled the VGA output and the background needs a redraw), or SIGUSR2 to force the selection of another wallpaper:


WP_RANDOM_PID=$(cat /tmp/wp-random_$DISPLAY.pid)

# Redraw current wallpaper
kill -SIGUSR1 "$WP_RANDOM_PID"

# Select another wallpaper
kill -SIGUSR2 "$WP_RANDOM_PID"