IOzone Performance Exploration, Part 2: The Rest of the Crowd (Almost)

In our previous Part 1 of this series we looked at the IOzone performance for ext3, ext4, and reiser4 using larger size record lengths (1MB, 4MB, 8MB, 16MB). The results are interesting because they unexpectedly showed some trends in performance that are independent of the file system. These observations are important if you take the results from a single record size run of IOzone and apply it to other record sizes (which you shouldn’t do).

In this article the number of file systems is extended to add most of the major file systems in Linux, even the experimental ones. This article adds ext2, jfs, xfs, btrfs, and reiserfs. As with the metadata benchmarks the purpose of this study is not to compare file systems and pick the “best” one (insert your definition of “best”). Rather, this study is an exploration of the performance of various Linux file systems using a single throughput benchmark. So the focus of this article is to explore how Linux file systems perform when IOzone is the benchmark.

Review of Last Results

This study uses IOzone to examine throughput of various access patterns for various Linux file systems. Recall that IOzone has a number of tests (access patterns) that stress the file system:

• Write
  
• Re-write
  
• Read
  
• Re-read
  
• Random Read
  
• Random Write
  
• Backwards Read
  
• Record Rewrite
  
• Strided Read
  
• Fwrite
  
• Frewrite
  
• Fread
  
• Freread
  

The general access pattern was discussed in the previous article.

A few observations were made in the previous article about the general behavior of the tested file systems:

• Write and re-write performance are relatively insensitive to record size over the range tested (1MB to 16MB) for the parameters of this test.
  
• Random write got much better as the record size increased from 1MB to 16MB. For example, ext3 went from 53,709.90 KB/s to 62,060.20 KB/s.
  
• Record re-write got worse as the record size increased. This performance can be driven by various cache and buffer sizes. Perhaps as the size increases, some cache sizes are exceeded and performance decreases.
  
• Read and re-read performance are relatively insensitive to record size over the range tested (1MB to 16MB) for the parameters of this test.
  
• Random read got dramatically better as the record size increased. For example at 1MB the random read performance for ext4 was 52,620.10 KB/s and at a record size of 16MB it was 105,651.60 KB/s. This is about a 100% increase in throughput performance! Some increase in performance is expected because there are fewer records as the record size is increased, but the change in performance is fairly dramatic.
  
• Backward read performance also dramatically improved with increasing record size. This is to be expected since there is less drive head movement because there are fewer total records.
  
• Strided read throughput performance also dramatically improved as the record size was increased. Again, it is about a 100% increase in performance from a record size of 1MB to a record size of 16MB.
  
• fread and freread are relatively insensitive to a change in record size.
  

Test System

The tests were run on the same system as the previous tests. The highlights of the system are:

• GigaByte MAA78GM-US2H motherboard
  
• An AMD Phenom II X4 920 CPU
  
• 8GB of memory (DDR2-800)
  
• Linux 2.6.30 kernel (with reiser4 patches only)
  
• The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ulta ATA/100)
  
• /home is on a Seagate ST1360827AS
  
• There are two drives for testing. They are Seagate ST3500641AS-RK with 16 MB cache each. These are /dev/sdb and /dev/sdc.

Only the first Seagate drive was used, /dev/sdb, for all of the tests.

IOzone Command Parameters

For this article, as with the last, only four record sizes are tested: (1) 1MB, (2) 4MB, (3) 8MB, and (4) 16MB. For a file size of 16GB that results in (1) 16,000 records, (2) 4,000 records, (3) 2,000 records, (4) 1,000 records.

The command line for the first record size (1MB) is,

./IOzone -Rb spreadsheet_output_1M.wks -s 16G -r 1M > output_1M.txt

The command line for the second record size (4MB) is,

./IOzone -Rb spreadsheet_output_4M.wks -s 16G -r 4M > output_4M.txt

The command line for the third record size (8MB) is,

./IOzone -Rb spreadsheet_output_8M.wks -s 16G -r 8M > output_8M.txt

The command line for the fourth record size (16MB) is,

./IOzone -Rb spreadsheet_output_16M.wks -s 16G -r 16M > output_16M.txt

Test Results

As mentioned previously, there are 13 tests that are each run 10 times and there are 4 record sizes. This makes a total of 520 tests that were run per file system.

Because of the large number of tests that are run, the results are split into two tables. The first table is for the write tests: write, re-write, random write, record re-write, fwrite, frewrite. The second table is for the read tests: read, re-read, random reads, backwards read, strided read, fread, and freread. Each group below has two tables for a specific record size group (1MB, 4MB, 8MB, 16MB). So that means there are 8 tables of results.

The first two tables of results are for the 1MB record size. Table 1 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 1 - IOzone Write Performance Results with a Record Length of 1MB and a File Size of 16GB

Table 2 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 1MB.

Table 2 - IOzone read Performance Results with a Record Length of 1MB and a File Size of 16GB

The next two tables of results are for the 4MB record size. Table 3 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 3 - IOzone Write Performance Results with a Record Length of 4MB and a File Size of 16GB

Table 4 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 4MB.

Table 4 - IOzone Read Performance Results with a Record Length of 4MB and a File Size of 16GB

The next two tables of results are for the 8MB record size. Table 5 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 5 - IOzone Write Performance Results with a Record Length of 8MB and a File Size of 16GB

Table 6 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 8MB.

Table 6 - IOzone Read Performance Results with a Record Length of 8MB and a File Size of 16GB

The final two tables of results are for the 16MB record size. Table 7 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 7 - IOzone Write Performance Results with a Record Length of 16MB and a File Size of 16GB

Table 8 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 16MB.

Table 8 - IOzone Read Performance Results with a Record Length of 16MB and a File Size of 16GB

There are a large number of tables with a great deal of data. These tables allow you to compare file systems but it may be a bit more difficult to see the trends as a function of record length. So the data in tables 1-8 can be tabulated for each file system. Then the performance for each test is tabulated as a function of the record size.

In general all of the file systems but btrfs have the same basic trend. Btrfs is slightly different. So only the data for reiser4, which is representative of all the file systems, and btrfs and are contained in the tables below. Table 9 contains the write throughput in KB/s for the 6 write tests for the various record lengths for reiser4.

Table 9 - IOzone Write Performance Results for Reiser4 with Various Record Lengths with a File Size of 16GB

Table 10 contains the write throughput results in KB/s for btrfs for the 6 write tests for various record lengths.

Table 10 - IOzone Write Performance Results for Btrfs with Various Record Lengths with a File Size of 16GB

Table 11 contains the read throughput results in KB/s for Reiser4 for the 7 read tests for various record lengths.

Table 11 - IOzone Read Performance Results for Reiser4 with Various Record Lengths with a File Size of 16GB

Table 12 contains the read throughput results in KB/s for Btrfs for the 6 read tests for various record lengths.

Table 12 - IOzone Read Performance Results for Btrfs with Various Record Lengths with a File Size of 16GB

For most of the tests, btrfs has the same trends as for the other file systems. For example, the trends for all of the read performance tests as a function of the record size is the same for reiser4 and btrfs. However, the random write performance test is a bit different. For reiser4 as the record size increases, the random write performance improves. This is to be expected since there are fewer writes and they increase in size as the record size increases. However, for btrfs, the trend is the opposite. As the record size increases, the random write performance of btrfs decreases. It’s not a large decrease, approximately 7.5%, but it is much more noticeable when reiser4 increases by approximately 28%. At this time, there is no reason for this trend in btrfs. But recall that btrfs is still under heavy development.

Observations

The observations have not changed too much with the addition of the other file systems. Most of the overall observations remain pretty much the same:

• Write and re-write performance are relatively insensitive to record size over the range tested (1MB to 16MB) for the parameters of this test.
  
• The random write performance gets better as the record size is increased from 1MB to 16MB. For example, jfs went from 53,182.10 KB/s to 108,949.50 KB/s, the greatest increase of all the file systems. However, the random write performance for btrfs remained about the same as the record size is increased.
  
• Record re-write performance decreased as the record size increased. This performance can be driven by various cache and buffer sizes. Perhaps as the size increases, some cache size are exceeded and performance decreases.
  
• Fwrite and frewrite performance are relatively insensitive to record size over the range of record sizes tested for the parameters of this test for all of the file systems tested.
  
• Read and re-read performance are relatively insensitive to record size over the range tested (1MB to 16MB) for the parameters of this test.
  
• Random read performance improved dramatically as the record increased. For example at 1MB the random read performance for ext4 was 52,620.10 KB/s and at a record size of 16MB it was 105,651.60 KB/s. This is about a 100% increase in throughput performance! An increase in performance is expected because there are fewer records, but the change in performance is fairly dramatic.
  
• Backward read performance also dramatically improved with increasing record size. This is to be expected since there is less drive head movement because there are fewer total records.
  
• Strided read throughput performance also dramatically improved as the record size was increased. For reiser4 the strided read performance increased from 49,071.90 KB/s at 1MB record size to 110,757.00 KB/s at 16MB.
  
• Fread and freread are relatively insensitive to a change in record size.
  

We are all human so it’s rather difficult to resist the temptation to compare the file systems. For the most part the file systems are tested with their default creation and mount options (ext3 and ext4 are mounted with “data=ordered”). So the above results may not show the file system in their best light. However, it may also reflect what many admins do - use the default options because they may not know the interaction between the options and the performance. At least many admins start with the default options and tune them based on testing. Therefore, drawing any conclusions from these test results about which file system is “superior” to the others is impossible. However, they can be contrasted.

There is quite a bit of data to be examined, but here are some observations:

• For write performance, most file systems are pretty good with the exception of ext3 and jfs. Jfs is by far the worst performing file system on this test compared to the top performing by about a factor of 2.5. Xfs is about the best, very closely followed by btrfs and reiserfs and then ext4, ext2, and reiser4.
  
• For rewrite performance, ext3 is the worst performing file system while jfs has about the best performance over the range of record sizes. Xfs, ext4, and reiserfs were close to the performance of jfs, but reiser4 and btrfs are not far behind.
  
• Reiserfs has about the best random write performance followed by btrfs, particularly at the smaller record sizes. At the larger record sizes, xfs and jfs have similar performance to btrfs. On the other hand, ext3, ext4, and reiser4 have about the worst performance.
  
• For record re-write, reiser4 is by far the best, particular at larger record sizes. The other file systems have about the same level of performance.
  
• For fwrite and frewrite, jfs is the worst performer by far - about a factor of 2. Ext3 is second from the bottom, fairly far behind the other file systems. The others are near or at the top in terms of performance but at smaller record sizes, xfs is the better performer.
  
• For read and re-read performance the ext file systems are behind the other file systems. Also, reiserfs has about the same performance as the ext file systems as well. Reiser4, jfs, xfs, and btrfs are the top performers and have about the same performance.
  
• For random read, ext3 has the worst performance - far less than the others. The others have somewhat similar performance to each other which is much better than ext3. Over the range of record sizes xfs seems to have the best performance, but at various record sizes, jfs, ext2, btrfs, reiserfs, and reiser4 are very close to xfs.
  
• For backwards read, ext3 is the slowest followed by btrfs. On the other hand, xfs and jfs have about the best performance but the others aren’t too far behind. In general reiserfs is just behind the others, but at the 1 MB record size it has the best performance by far.
  
• In the case of strided read, ext3 is the slowest by a reasonable margin. Near the top are jfs and xfs.
  
• For fread, ext3 has the worst performance but ext2 is fairly close. Xfs, jfs, and reiser4 are about the best performers but btrfs and reiser4 are very close. Ext4, reiserfs, and ext2 are a little further back.
  

Based on these observations I think it’s difficult to declare an overall “winner”. Several of the file systems perform very well over the range of tests and others don’t perform well in certain tests.

Next Time

One file system that is missing from this article is nilfs2. Recall that nilfs2 is a log-structured file system which is different in design than the others. It also has the feature of creating checkpoints after each sync system call. IOzone does a great deal of syncs during it’s tests. Consequently, nilfs2 is a bit more difficult to compare to the other file systems.

In an upcoming article we’ll test nilfs and explain the testing methodology using IOzone. It should be interesting so keep an eye out for this article.