Watch your inodes !

Today I got a strange bug doing apt-get update which gave me 404 error for each repository. At first I though it was the apt package which ran out-of-date but when I tried to install it manually dpkg said something like : “No space left on device”. Trying to touch something achieved the same result. So I did :

# df -h /var

But it said that about 1GB were available on that filesystem. Then :

# tune2fs -l /dev/sda11 | grep “Free”
Free blocks: 276171
Free inodes: 0

There were no inodes availables on the filesystem. I always though that filesystem have enough inodes in it so we don’t have to care about that but actually squid created many small files, worrying about blocks usage but not about inodes usage. I got it fixed by deleting the squid’s cache directory and setting squid.conf to use less top-level and sub-level directories.

About ext4

The ext4 is a journaled file-system, the successor to ext3. In summary the ext4fs greatly improve overall performance, efficiently reduce fragmentation and is fscked much more faster than his predecessor. Actually it has many features which follow :

  • Large file system with now 16TB files and up to 1EB volume size.
  • Extents which reduce fragmentation and improve large file performance.
  • Compatibility with ext2 and ext3.
  • Preallocation.
  • Delayed allocation which improve block allocation decisions.
  • Break 32 000 subdirectory limit with now 64 000 and improved performance using htree by default.
  • Journal checksumming which improve reliability.
  • Online defragmentation (didn’t try yet).
  • Multiblock allocator which reduce fragmentation.
  • More precise timestamp and fix the 2038 bug.

If you want to create an ext4 file-system first get the linux kernel (>= 2.6.28) which contains the first stable version of the filesystem. Then enable CONFIG_EXT4_FS as module or built-in. If you want to try the large files and volume feature add CONFIG_LBD actually I didn’t try it since the largest I’ve got here is about 2TB so in what follows I’ll assume you won’t enable this feature. Then install e2fsprogs (>= 1.41) and check in /etc/mke2fs.conf for a line looking like this :

ext4 = {
features = has_journal, extents, huge_file, flex_bg, uninit_bg, dir_nlink, extra_isize
inode_size = 256
}

Checks the features and inode size (should be at least 256) and remove the huge_file feature since we won’t use it. Now reboot on your new kernel, create the partition with fdisk or any other tools and let /dev/sdb2 be that partition. Just do as root :

# mkfs.ext4 -I 256 -m 0 -L HOME /dev/sdb2

This will create an ext4fs on /dev/sdb2 with 0% of reserved blocks for root labelled as HOME (always label your file-systems it makes it easier to manage your partitions) and enable every features found in /etc/mke2fs.conf on it. Give it a try and mount it on /mnt try putting some file on it :

# mount -t ext4 /dev/sdb2 /mnt

Now in order to transfer your files to the newly created file-system just do :

# mount -t ext4 /dev/sdb2 /mnt && cp -aRv /home/* /mnt/ && rm -rf /home/* && umount /mnt && mount -t ext4 /dev/sdb2 /home

Your home directories should now be mounted on the new ext4 file-system. Be sure you didn’t misstyped the last command since it will delete files on the old file-system. Then just add a line for it in /etc/fstab in order to mount it automatically on each boot :

/dev/sdb2 /home ext4 noatime 0 2

Now your home directory will be mounted automatically. Repeat that step for each partition and it’s done.

The ext4 file-system is not shown in the squeeze’s debian installer so there are three solutions to take advantages of ext4 on the current stable :

  1. Using an ext3 partition and then mount it as ext4 using backward compatibility. I don’t use it since I heard that some features won’t be enabled that way especially the extents feature won’t if the inode size of the ext3 file-system is not at least 256.
  2. Using multiple ext4 partitions on top of an ext3 root instead of a single partition. This is the simplest solution which I’m using on my laptop right now.
  3. Tweaking the debian installer to install the system on a manually created ext4 file-system. This is the most boring solution since if it fails somewhere you have to restart the installation process from the beginning.

I’ll just describe briefly the second solution here since I don’t like the first one neither do I remember every step of the third one. When installing debian create two partition one for /boot (~200MB) and another for the root, I mean “/” (~1GB) flagged for use as ext3 or reiserfs (actually I’m using reiserfs). Then install only the base-system. Rebuild a new kernel with ext4 enabled in it, create partitions for /var, /usr, /tmp, /home and transfer documents as described above. Now continue the installation using aptitude and that’s it.