Ext4 on Wimpy's World

Give XFS a chance. Don't believe the FUD.

martin@wimpress.com (Martin Wimpress) — Fri, 12 Feb 2010 13:14:44 +0000

After tinkering with Ext4 I did some research and tested other file systems on my new disk arrays. I’ve concluded that XFS, once tuned, is the best file system for my needs and it could well be the best file system for your needs too.

The remainder of this page explains how I arrived at that decision and how I tune XFS to get optimal, yet safe, performance that can rival Ext4 and JFS.

Benchmarks

Here are some benchmarks.

Why I chose XFS

I have not chosen XFS for performance alone, indeed some benchmarks show that XFS it outperformed for some file operations.

My workstation at home has two 6TB disk arrays and a 1TB root file system. The disk arrays contain photo, music and video libraries which are streamed via UPnP/DLNA and DAAP. The video files can be 2GB to 30GB in size. I also do a good deal of HD video encoding, processing and editing. My root partition contains many virtual machine images of which several are running at any given time.

My work laptop has a 250GB root file system and also contains many virtual machine images of which one is usually running.

XFS is designed with large file systems and large file handling in mind. It seems a sensible choice for those reasons alone, but I also liked the following features:

XFS has on-line defragmentation tools, while (at the original time of writing in early 2010) Ext3/4 and JFS do not.
XFS dramatically reduces start-up time by avoiding fsck delay. Ext3/4 can be very slow to fsck large volumes.
XFS has very fast (a few seconds or less) file system creation. JFS is faster than XFS but Ext4 takes many, many minutes.
XFS formatted disk capacity is greater than Ext3/4 even after removing the reserved blocks from the Ext3/4 file system. JFS formatted capacity is similar to XFS.

On that last point, XFS gains 400GB over Ext4 on a 6TB array but when the Ext4 reserved blocks are removed XFS gains 100GB over Ext4.

Tuning XFS

Most of the performance tuning information I found (at the original time of writing in early 2010) is out of date and doesn’t reflect the XFS defaults in modern Linux kernels.

That said, the information on this page is quite old and I no longer feel the need to tweak XFS like I once did.

Creating XFS

XFS 3.1.0 and Kernel 2.6.32 or newer

Ubuntu Lucid 10.04 comes with XFS 3.1.0. The defaults used when creating a XFS file system using Ubuntu 10.04 are optimal and do not require any tweaking.

XFS 3.0.2 and Kernel 2.6.31 or older

Ubuntu Karmic 9.10 comes with XFS 3.0.2. If you are running an earlier Ubuntu release and want to use a tuned XFS root file system you can’t simply use the graphical partitioning tool from the Ubuntu LiveCD installer. However, it is very easy manually create the tuned XFS file systems. Simply boot the Ubuntu Live CD, then start a new shell Application -> Accessories -> Terminal.

Now run the following as root.

mkfs.xfs -l lazy-count=1 -L VolumeName <dev>

lasy-count=1 is a default since XFS 3.1.0 but was recommended by the XFS developers before that. lazy-count is a mkfs option because it changes the on-disk format slightly, and older kernels do not understand this new format. Hence mkfs sets a superblock feature bit to prevent the file system from being mounted on kernels that don’t understand the slightly different disk format. So you must specify lazy-count=0 if you want to disable this feature for older kernels which don’t support it.

http://oss.sgi.com/archives/xfs/2007-12/msg00536.html

Forcing a tuned XFS creation

If you are not sure what XFS version you are running, and therefore what the defaults might be on your system, you can fully tune XFS using the following.

For < 1TB XFS file system

mkfs.xfs -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4 -L VolumeName <dev>

For > 1TB XFS filesystem

mkfs.xfs -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=16 -L VolumeName <dev>

Once you have created all your tuned XFS file systems start the Ubuntu installer from the Live CD. When the disk partitioning section comes round choose: Specify Partitions Manually

Now Change each XFS file system telling the partitioner where to mount each XFS file system. But ensure that you do not tick Format the Partition:, thereby preserving your tuned XFS file systems.

When you see this message, just click Continue.

The file system on /dev/sda1 assigned to /boot has not been marked for formatting. Directories containing system files (/etc, /lib, /usr, /var, …) that already exist under any defined mountpoint will be deleted during the install.

Please ensure that you have backed up any critical data before installing.

Mounting XFS

Further performance optimisations can be gained but specifying some additional mount options for your XFS file systems.

To manually mount a XFS file system with, optimal mount options, use the following:

mount -t xfs -o noatime,osyncisosync,logbsize=256k,logbufs=8 <dev> <mtpt>

The /etc/fstab entries I use look something like this.

UUID=xxxxxxxxxxx...x <mtpt> xfs noatime,osyncisosync,logbsize=256k,logbufs=8 0 2

The logsbsize' and logbufsoptions address the often sited limitation of XFS when handling lots of small files and large number of file deletions. The above assumes you don't requireatime. Not using atime` provides a significant performance benefit.

atime, relatime and noatime

Every time a file is accessed (read or write) the default for most file systems is to append the metadata associated with that file with an updated access time. Thus, even read operations incur an overhead associated with a write to the file system. This can lead to a significant degradation in performance in some usage scenarios. Appending noatime to the fstab line for any file system stops this action from happening.

One may also specify a relatime option which updates the atime if the previous atime is older than the mtime or ctime. In terms of performance, this will not be as fast as the noatime mount option, but is useful if using applications that need to know when files were last read (like mutt).

As access time is of little importance in most scenarios, this alteration has been widely touted as a fast and easy way to get a performance boost. Even Linus Torvalds seems to be a proponent of this optimization

http://kerneltrap.org/node/14148

Access time is not the same as the last-modified time. Disabling access time will still enable you to see when files were last modified by a write operation.

async and nobarrier

If you really want to go for all out performance you can also provide async and nobarrier mount options. But you really need to understand and accept the potential issues with using these options.

Read the following to understand what write barriers are and if you are prepared to disable them to gain performance.

http://lwn.net/Articles/283161/

XFS userspace tools

XFS is available as a kernel module in Ubuntu and also available from the Live CDs. Once Ubuntu is installed you can install the XFS userspace tools as follows.

sudo apt-get install xfsdump xfsprogs

De-fragmenting XFS

There are two utilities that XFS has to manage this fragmentation.

xfs_db XFS Debug Information. Used to examine an XFS file system for problems or gather information about the XFS file system.
xfs_fsr File System Organiser. Improves the organisation of mounted file systems. The reorganisation algorithm operates on one file at a time, compacting or otherwise improving the layout of the file extents (contiguous blocks of file data).

Defragment a file system

To find the health of a XFS file system use the xfs_db command to gather some information. In the example below /dev/sda1 is mounted as /boot and /dev/sda3 is mounted as /root.

sudo xfs_db -c frag -r /dev/sda1
actual 162, ideal 162, fragmentation factor 0.00%

sudo xfs_db -c frag -r /dev/sda3
actual 2288833, ideal 254504, fragmentation factor 88.88%

The closer the fragmentation factor is to 0% the better. Unsurprisingly /boot is not fragmented. However /root is very fragmented.

Defragmenting XFS file systems can be done on a live running system, but it is a good idea to schedule this for a time where the partition will be used less.

The file system reorganizer for XFS is xfs_fsr. Typically, I instruct xfs_fsr to reorganise /dev/sda3 with a timeout (-t) of 6hrs (60 * 60 * 6 = 21600) which is specified in seconds. But for the purposes of this example I used a timeout of 15 mins.

sudo xfs_fsr -t 300 /dev/sda3 -v

The output will look something like this.

/ start inode=0
ino=145565
extents before:2 after:1 DONE ino=145565
ino=145662
extents before:2 after:1 DONE ino=145662
ino=600148
extents before:2 after:1 DONE ino=600148
ino=1127295
extents before:82794 after:1 DONE ino=1127295
ino=1127243
extents before:2 after:1 DONE ino=1127243
ino=1382852
extents before:50869 after:1 DONE ino=1382852
ino=1422636

When the defrag is finished check how well the file system reorganising was.

sudo xfs_db -c frag -r /dev/sda3
actual 2155648, ideal 254512, fragmentation factor 88.19%

As you can see defragmenting for 15 mins doesn’t improve things greatly, which is why xfs_fsr needs to be run for several hours or more.

Manually defragmenting the file system is simple enough, but a better solution would be to schedule a cron job to run periodically.

Defragment a file

It is also possible to de-fragment a single file. To determine if a file is in need of defragmenting run the following…

xfs_bmap -v /srv/A320/PGQAR.DAT | wc -l

This will output a number which showing the number of extents the file is using.

This number should be close to 1. So in the example above, I have a very fragmented file.

sudo xfs_fsr -v /srv/A320/PGQAR.DAT

This will output something like the following.

/srv/A320/PGQAR.DAT
extents before:95278 after:1 DONE /srv/A320/PGQAR.DAT

The file is now defragmented. I use the method above to target defragmentation where I know files reside that are most likely to be fragmented, rather than defragmenting the whole file system.

References

XFS References

Performance Tuning XFS References

Recovering reserved space from ext4

martin@wimpress.com (Martin Wimpress) — Thu, 07 Jan 2010 20:42:04 +0000

The Ext4 file system, like Ext3, reserves 5% of the blocks on the file system for the root user. The reserved blocks are there for root’s use as a safe guard if the filesystem gets full, it provides some wiggle room to enable the really important programs to still function. But in some cases there’s not much point in having space reserved for root. I’ve recently upgrade my workstation with a 6TB internal RAID 0 array for data storage (music, videos, photos, etc) and an external 6TB RAID 0 array as a backup. My OS boot from a 1TB drive. For my 6TB arrays I want the maximum available storage and was interested to see what effect removing the reserved space would have. So, this is what I did. First I made the Ext4 file system, mounted it and queried how much space was available.

sudo mkfs.ext4 /dev/sdh1
sudo mount /dev/sdh1 /mnt
df -h

Looks like I have 5.1TB of available space.

/dev/sdh1             5.4T  186M  5.1T   1% /mnt

Then I unmounted the file system, removed the reserved blocks, checked the consistency of the file system, mounted it and queried how much space was available.

sudo umount /mnt
sudo tune2fs -m 0 /dev/sdh1
sudo e2fsck /dev/sdh1
df -h

Looks like I have 5.4TB available now, a saving of 300GB.

/dev/sdh1             5.4T  186M  5.4T   1% /mnt

Now, I could have simply created the files system without the reserved blocks in the first place, but I was interested to see the comparison.

sudo mkfs.ext4 -m 0 /dev/sdh1

Before you start removing the reserved blocks from your ext3/ext4 file systems do a bit a research first.

StarTech S354UER Review

martin@wimpress.com (Martin Wimpress) — Sat, 02 Jan 2010 16:00:25 +0000

*** UPDATE: The StarTech S354UER completely died after less than one year. Not recommended! ***

I’ve ripped my entire CD collection to MP3 and I’m in the process of ripping my entire DVD, Blu-Ray and HD-DVD collection to MPEG-2 TS files so that I can stream everything to my PS3 using MiniDLNA. I currently have this data stored on an internal 2TB volume and backed up to an external 2TB volume. I currently have just 360GB remaining capacity and I’ve only imported half my DVD collection and one Blu-Ray. I need more storage.

I wanted to keep the same backup method, large internal volume backed up to a large external volume of the same size. I opted for Samsung Spinpoint F2 EcoGreen (HD154UI) drives because they are relatively inexpensive, low power (therefore lower heat) and quiet.

I decided to get 4x 1.5TB drives for the internal volume and stripe them to give 6TB of storage and I went looking for an external box in which I could install 4x 1.5TB drives and also stripe or span them. That meant a multi disk external enclosure, with some kind of RAID, supported by Linux and that isn’t too expensive. A tall order as it turns out.

After lots of research I finally found the StarTech S354UER which on paper appears to do what I required and a good deal more.

Compatible with Windows, Mac, and Linux operating systems
Fan control button to enable manual control of the fan and switch between the three fan speeds
Internal three speed 80mm fan with automatic or manual controls
Multiple LED indicators to provide RAID information, hard drive activity, HDD Status, RAID rebuild status, fan settings, and interface in use
No software required
Package includes 1x USB, 1x eSATA, 1x FireWire 400, 1x FireWrie 800 cable, Power adapter and cord, and the manual
Plug-and-Play and Hot swap supported with USB 2.0, eSATA, and FireWire
Push button raid configuration eliminated the need to disassemble the enclosure to upgrade your raid configurations
Removable front cover for easy access to hard drive
Rugged aluminum chassis
Supported File Systems:NTFS, FAT, FAT32, and ext3
Supports four 3.5in hard drives up-to 2.0 TB each in size
Supports RAID 0, RAID 1, RAID 3, RAID 5, RAID 10 (RAID 1+0), and Spanning

Normally, I will read reviews of different products and select something with a proven track record particularly when Linux support is required. I couldn’t find much in the way of reviews for the StarTech S354UER so I took a gamble and decided to buy one. Eeek!

In short it works and it is quiet. It is currently sitting no more than 50cm from me initialising a stripped array of 4x1.5Tb disks as Ext4. I can’t hear it but I have manually set the fan speed to low using the fan control buttons on the front on the chassis.

The build quality is not great, but not awful either, but once the drives are installed and clamped in place they are very secure. It is impossible to tell if you’ve pushed the power button you have to wait and see if the device powers up/down to be sure. The fan speed controls work, but are inverted from what is documented in manual. Fan1 is documented as LOW in the manual but is actually HIGH.

Setting up the device was not quiet plug and play either but the issues I ran into may not be entirely the fault of the StarTech S354UER. My plan was to connect the enclosure via Firewire and as yet I’ve not been able to get the enclosure to be recognised via Firewire using Ubuntu Jaunty 9.04. However, I am a Firewire newbie so maybe more research required. I don’t have eSATA (yet) so I have the device connected via USB 2.0. Which does work.

One of my new hard disks turned out to be DOA. It took me a while to figure out what was wrong here. The StarTech drive failure light on the front of the chassis was illuminated, but I didn’t know how to tell which drive had actually failed. After some trial and error I found that there are four internal LEDs, one for each disk. Starting the StarTech with the chassis door open you can see the internal LEDs blinking as each disk is spun up and tested. If the drive failure LED on the front of chassis is illuminated look at the internal LEDs, the drive LED which is off denotes the failed drive. This information is not in the user manual!

I replaced the drive and was able to select my RAID level. Selecting the RAID level is done though a combination of DIP switches under a panel at the back of the unit and buttons on the front. It is a slightly fiddly process, but it does have the advantage that you can’t accidentally change your RAID levels and re-initialise the array.

In order to create a partition greater than 2TB you have to use GPT. I’d not encountered GPT before, but I found everything I needed to know on the page below.

Make the most of large drives with GPT and Linux

I used gparted to create my partition and format with Ext4 with the enclosure connected via USB 2.0. The whole process took ~1 hour. As you will see from the link the quoted price makes this enclosure pretty expensive. Search around though, because I got mine of 50% less than the price quoted on the StarTech.com website. You do get a healthy selection of RAID levels, all the cables, screws, screwdriver and drive handles you require.

Would I recommend the StarTech S354UER? Time will tell, but it does work with Linux via USB 2.0 and I will continue investigate FireWire and I may add eSATA in the future to see what the performance benefits are. But for what I bought it for, secondary storage for backups, it is a pretty cheap way to add a multi terrabyte array to your system.