FS-Cache & CacheFS: Caching for Network File Systems

When *nix OS’s were developed, systems could be a bit on the slow side. Typical networks were either 10 Mb/s or, if you were lucky, 100 Mb/s. Accessing network based file systems such as NFS and AFS could be rather slow over these networks. In response to sometimes slow access, a local caching mechanism, called CacheFS, was developed to provide local caching for distributed file systems. This mechanism caches data (either complete files or parts of files) on local storage so that data can be possibly accessed from a local storage device instead of a network based file system.

As part of the 2.6.30 kernel bonanza of file systems, a CacheFS implementation for Linux was added. Currently it supports NFS and AFS, but other file systems could benefit from such a caching mechanism (more on that later in the article). This article is a brief introduction to CacheFS, which is still undergoing a few growing pains, but is showing great progress for helping network file system performance with an eye toward other uses. In addition, the article will discuss FS-Cache which is a thin layer between CacheFS and the file system itself.

What is FS-Cache and CacheFS?

If you have read about FS-Cache or CacheFS you can easily get confused by the terminology. FS-Cache is the interface between the file system and the cache, allowing the file system to be cache agnostic. CacheFS is the caching backend for FS-cache. CacheFS does the actual handling of data storage and retrieval (i.e. the caching mechanism) and uses a partition on a block device. There is also an alternative caching mechanism called CacheFiles that uses a directory in an existing file system. This architecture allows alternative caching mechanisms to be used with FS-Cache. The first implementation of CacheFS was developed by Sun in 1993 for the Solaris 2.3 operating system. After that, other versions soon appeared, notably for Irix in 1994.

A number of years ago, some Linux kernel developers, notably Dave Howells and Steve Dickson, began to develop FS-Cache and CacheFS for Linux. In August 2004, Dave posted a set of patches to the Linux kernel mailing list for CacheFS. FS-Cache patches have been posted along with CacheFS. A good overview of CacheFS is an article on lwn.net. With the 2.6.30 kernel, CacheFS is included and is available for NFS and AFS.

CacheFS cannot be used for just any file system. The file system must be written to use FS-Cache. FS-Cache can use any caching mechanism it wants (using the caching interface) and the file system does not care. Both AFS and NFS have been modified to use FS-Cache. The 2.6.30 kernel contains these modified versions that utilize FS_Cache.

The diagram below gives you a general view of the data flow when using FS-Cache and CacheFS.

Figure 1: Data Flow showing FS-Cache, CacheFS, and CacheFiles

In the diagram, NFS or AFS or ISOFS, which are generically called the netfs, make function calls to FS-Cache which in turn calls either CacheFS or CacheFiles. The reason for FS-Cache is to allow the file systems on the left hand side to use caching without having to know about the implementation. This keeps the API a constant for the file systems (NFS, AFS, etc.) while allowing developers to slip in different caching schemes as needed (or desired). Notice that CacheFS uses a partition on a block device, in the diagram this is /dev/hda5, while CacheFiles, uses /var/fscache by default to store the locally cached files.

Figure 2 below shows a bit more detail for the data flow of netfs and FS-Cache.

Figure 2: More Detailed Data Flow showing FS-Cache, CacheFS, and CacheFiles along with netfs

In this diagram, it shows how the netfs file system communicates with FS-Cache, which in turn communicates to CacheFS or CacheFiles. Notice too that the netfs also communicates with the VFS as needed (CacheFS is just for caching and is not intended to be a complete file system by itself. Consequently, accessing the VFS by the netfs is necessary). The kernel documentation for FS-Cache is here and the kernel documentation for the api for the netfs is here. Both provide a wealth of information.

The key concept behind FS-Cache is that it does not itself require a file from the netfs to be completely loaded into the cache before the file can be accessed through the cache. This is because:

• It must operate without a cache


• It must be possible to open a remote file that’s larger than the cache


• The combined size of all open remote files must not be limited to the size of the cache


• The user should not be forced to download an entire file just to access a portion of the file for one time
  

Consequently, FS-Cache is built around the concept of pages. So pages, rather than files, are put into the cache and are used when the netfs requires it. The FS-Cache implementation in Linux provides the following features (from the kernel documentation):

• More than one cache can be used. They are identified by tags to allow them to be individually identified


• A cache can be shared between multiple netfs. In the event that they are shared, the cached data of each netfs is independent of the other. In addition, it does not associate two different views of the same file. For example, this means that if a file is read by both NFS and CIFS (if CIFS is FS-Cache aware), then two copies of the file will end up in the cache.


• The caches can be removed or added at any time


• The netfs is provided with an interface that allows either party (netfs or FS-Cache) to withdraw caching facilities from a file


• The interface to the netfs returns as few errors as possible, preferring to let the netfs remain oblivious


• Data IO is done directly to and from the netfs’s pages


• As much as possible is done asynchronously

Dave Howell has also published a paper that discusses FS-Cache more in-depth. It also talks about CacheFS and CacheFiles.

Just as with everything in life, there are some positives and negatives to things. FS-Cache and CacheFS follow this rule of thumb. Dave Howells has done a good job explaining what the trade-offs are in terms of performance. Summarizing from his paper:

• File Lookup Time will be:




• Increased because the cache will be checked before resorting to the network.


• Increased because the a looked-up object in the cache must be noted.


• Decreased because the metadata is cached locally




• File Read Time will be:




• Increased because the cache much be checked before resorting to getting the data over the network


• Increased because the data obtained must be copied back to the cache


• Decreased because of the local caching which should be faster than going over the network




• File Write Time will be:




• Decreased because of writeback caching using the disk


• Neutral because write-through caching should be able to write to the network and the cache at the same time




• File Replacement Time will be:




• Increased because an object or tree of objects will have to be retired from the disk

The performance of the network and the server are also affected. The goal of FS-Cache and CacheFS is to reduce network traffic because some of the data requests will be satisfied by local storage (CacheFS) reducing the amount of network traffic. The load on the server should also be reduced since it will not have to satisfy all data requests. Consequently, this reduction may make up for the increased file lookup time and file read time due to the cache.

Keep in mind that the local caching might not produce an overall throughput – it’s all up to the particular load and IO pattern. It is not well suited for random access IO such as that from databases because it is unlikely that the data will be in cache. But for IO patterns that are more sequential in nature, CacheFS could help performance.

Moreover, files that are opened with O_DIRECT should not go through the cache. It is up to the netfs to take care of this so that FS-Cache shouldn’t even be contacted. But if your application uses this flag when opening files, caching may not work as well as you had hoped (it shouldn’t work at all).