Harping on Metadata Performance: New Benchmarks

Metadata performance is perhaps the most neglected facet of storage performance. In previous articles we've looked into how best to improve metadata performance without too much luck. Could that be a function of the benchmark? Hmmm...

If you have been reading the articles in this column for any amount of time you will see a number of them focus on metadata performance. The primary reason for focusing on metadata performance is that it is one of the most critical aspects of overall performance and is one of the most neglected aspects. When people talk about drive or storage performance it is invariably given in terms of MB per second or something similar. In other words – throughput. However, people do not realize how important metadata performance is to overall performance.

Metadata performance refers to the how quickly files and directories can be created, removed, their status checked (stat), as well as other data functions. This aspect of storage performance is becoming more important because of the increasing number of files and directories on systems. Creating files, deleting them, and performing a status check of them is important for more applications than ever before. There are applications that can produce millions of files in a single directory and applications that create very deep and wide directory structures. As the number of cores increases the number of files and number of directories increases putting more and more pressure on the metadata performance of storage solutions.

In previous articles a simple metadata benchmark, fdtree, was used to measure benchmark performance. We saw that many times there was not a lot of change in performance when the file system metadata performance was changed. This led me to ask the question, is fdtree a good enough benchmark to show metadata performance as one tweaks a system for performance? If you like analogies, it’s like wondering if you doctor has the right diagnosis. Since I don’t always trust single opinions, I always like to get second and even third opinions. So in this article I want to examine a new metadata performance benchmark, run it on the same hardware as past articles that used fdtree, and see if they both show the same trends.


A common benchmark used for HPC storage systems is called metarates. Metarates was developed by the University Corporation for Atmospheric Research (UCAR). It is a MPI application that tests metadata performance by using POSIX system calls:

  • creat() – (open and possibly create a file)
  • stat() – (get file status)
  • unlink() – (delete a name and possibly the file it refers to)
  • fsync() – (synchronize a file’s in-core state with storage device)
  • close() – (close a file descriptor)
  • utime() – (change file last access and modification times)

Using these system calls, the main analysis options for metarates are the following:

  • Measure the rate of file creates/closes (create/close per second)
  • Measure the rate of utime calls (utimes per second)
  • Measure the rate of stat calls (stats per second)

Metarates has options for the number of files to write per MPI process (remember that you will have N processes with a MPI application where N is a minimum of 1) and if the files are to be written to a single directory or to many directories. It also has the option of using the system call fsync() to synchronize the file’s in-core state with the storage device. But fsync has been something of a controversial subject in Linux so let’s cover fsync() for a moment.

The system call fsync is really designed to flush the buffer cache data to the storage devices (e.g. disks) to make sure that the data is actually on the disk and not in a buffer where a power lose would mean losing data. According to the eminent Theodore Ts’o fsync() does the following:

"... the only safe way according that POSIX allows for requesting data written to a particular file descriptor be safely stored on stable storage is via the fsync() call. ..."

However, if you read the man pages for fsync() (“man 2 fsync”), you will see the following,

"Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync() on a file descriptor for the directory is also needed."

So fsync appears to be the best way for an application to ensure that the data has actually reached the disk but you still have to be careful if you want to be 100% sure that the data is actually on the disk.

If you want to try a simple experiment examine the IO output of your applications and look for calls to fsync. I think you would be surprised at how many applications do not use use it.

Test System

The test system used for these experiments was a stock CentOS 5.3 distribution but with a 2.6.30 kernel and e2fsprogs was upgraded to 1.41.9. The tests were run on the following system:

  • GigaByte MAA78GM-US2H motherboard
  • An AMD Phenom II X4 920 CPU
  • 8GB of memory
  • Linux 2.6.30 kernel
  • The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ulta ATA/100)
  • /home is on a Seagate ST1360827AS
  • There are two drives for testing. They are Seagate ST3500641AS-RK with 16 MB cache each. These are /dev/sdb and /dev/sdc.

Only the first Seagate drive was used, /dev/sdb for all of the tests. The second drive, /dev/sdc was used only for the tests where the journal was placed on a drive.

As with previous articles, ext4 is used as the file system. The journal can be easily moved to a different device so that the impact of the journal location on performance can be determined. The steps for creating the file system and the journal are covered in previous articles.

Metarates Command Parameters

The test system has 4 cores and Metarates is an MPI application allowing us to choose the number of processes (cores) we use during the run. So for this benchmark 1, 2, and 4 cores were used (three independent tests).

Not forgetting our good benchmarking skills, the run time (wall clock time) of the runs should be greater than 60 seconds if possible. So the number of files was varied for 4 MPI processes until a run time of 60 seconds was reached. The resulting number of files from the test was found to be 1,000,000 and was fixed for all tests. Also it was arbitrarily decided to have all files are written to the same directory with the goal of really stressing the metadata performance.

The final command line used for metarates for all three numbers of processors (1, 2, and 4) is the following.

time mpirun -machinefile ./machinefile -np 4 ./metarates -d junk -n 1000000 -C -U -S -u >> metarates_256MB_ramdisk.np_4.1.out

Where the “-np” option stands for number of processes (in this case 4), “-machinefile” refers to the list of hostnames of systems to be used in the run (in this case it is a file name “./machinefile” that contains the name of the test machine repeated 4 times – once for each process), and the results to stdout are sent to a file “metarates_256MB_ramdisk.np_4.1.out” which is an example of how the output files were named.

Notice that fsync wasn’t used in the benchmark run so that the results would be comparatively similar to those of fdtree which didn’t explicitly fsync() files. In addition, the directory where the files are written is the current directory where the binary (“./metarates”) is located. Also notice that three different performance measures are used:

  • File create and close rate (how many per second)
  • File stat rate (how many “stats” per second)
  • File utime rate (how many “utimes” per second)

Next: Running Metarates

Comments on "Harping on Metadata Performance: New Benchmarks"

If you want to improve your knowledge only keep visiting this website and be updated with
the most recent news posted here.

I think this is one of the most vital information for me.
And i’m glad reading your article. But wanna remark on few general things, The website style is wonderful, the articles is really excellent :
D. Good job, cheers

Link exchange is nothing else however it is just placing the other person’s weblog link
on your page at suitable place and other person will also
do similar for you.

Regardless of all the upcoming massive claims and
hype, this item has the prospective to make all other forex manuals appear like hobby guides.

Also visit my blog post web site (Valerie)

Hi, this weekend is good in favor of me, as this occasion i am reading this
fantastic informative paragraph here at my home.

This is a good tip especially to those fresh to the blogosphere.
Short but very precise info… Thank you for sharing this one.
A must read article!

Can I just say what a relief to discover somebody
that truly understands what they’re discussing online. You actually understand how
to bring a problem to light and make it important.

A lot more people should look at this and understand this side of
the story. I was surprised that you’re not
more popular since you definitely have the gift.

I went for a excеptional gift аnd ⅼеarneԀ
tɦіs рɑіntіng tɦɑt ⅼіǥһtіng insіԁе tһе dіm.

Ιtѕ ѕߋmᥱкіnd
оf ɗօսaⅼ-νiеѡ ρaіnting.
І саn ⅼоοк ɑt іt aⅼⅼ tҺгоᥙցһ tһе
ѡⲟrҝіng ɗɑy, bսt ɑftᥱг is
ⅾaгқіsɦ іn tҺᥱ һoᥙѕе, tһе рогtгɑʏ іs neνегtһеⅼeѕѕ аρрaгent.

Is tһiѕ ѕоmе foгm оf alіеn tеϲҺnoⅼⲟցy/ ⲟг іѕ simρlу juѕt sοmе еⲭсеρtіօnal ᴡаll artᴡߋrқ?

Ноԝᥱveг, і Ԁеtеrmіneⅾ
tоաaгɗ cɦоⲟѕe it aѕ а ǥіft,
ѕtіⅼⅼ іn dігectіοn ⲟf acԛᥙіrе ѕⲟme fοr myѕeⅼf mսcһ tߋⲟ :).

I fᥱeⅼ tɦаt іnsidе οf maкᥱ
any Ԁiffeгеncе οf ρгеsеntѕ, tһіѕ ѡоuⅼⅾ
bе tҺᥱ ցгᥱatеѕt.

Ɗο օneѕelf ѕensе thе vᥱгү sɑmе?


Hеllo veгy ϲօοⅼ bⅼߋց!!
Ꮐᥙy .. Εхcelⅼеnt ..
Ѕᥙpеrƅ .. І աіll
bοοкmarқ үօᥙг ѡеbѕitе аnd tɑқᥱ tһᥱ
feᥱԀѕ ɑⅼѕo?
I’m ѕatiѕfіᥱԀ tߋ sеɑгϲh ⲟᥙt sο mаny սѕefuⅼ іnfoгmatіοn hеге ᴡіtһіn tҺе рoѕt,
ᴡe’ɗ ⅼікᥱ dеѵelоⲣ ᥱⲭtгa teϲһniqսеѕ іn tҺіѕ rеgагԀ, tһankѕ fօr
sһarіng. . . . .

Mу ⲣagе: ???????????

I just want to say I am just new to blogging and definitely loved your website. Likely I’m want to bookmark your blog post . You certainly come with tremendous stories. Thanks a bunch for sharing your blog site.

A person necessarily help to make severely articles I would state.

I just want to tell you that I am all new to blogging and site-building and really loved this blog. Likely I’m planning to bookmark your website . You actually come with fabulous articles. Many thanks for sharing with us your website.

This will save you hundreds on the designer bags that you are after.

Generally speaking, the bags are either a bit plain, insanely expensive, or more frequently both.
The budget for buying bags depends from woman to woman.

my homepage: cheap prada bags

I like your site and content. thanks for sharing the information keep updating, looking forward for more posts. Thanks

I like your site and content. thanks for sharing the information keep updating, looking forward for more posts. Thanks
selbstklebende bh

I just like the helpful information you provide to your articles.
I’ll bookmark your blog and test once more right here frequently.
I’m slightly sure I will learn many new stuff proper
right here! Good luck for the following!

Also visit my website – Dean Graziosi

I just want to tell you that I am just all new to blogging and site-building and honestly enjoyed you’re blog site. Most likely I’m want to bookmark your blog . You surely have really good well written articles. Thanks a bunch for sharing your website page.

Feel free to surf to my website … Lenora

Feel free to surf to my webpage :: Joie

for does results. supplement. the medicine the but to as may the forskolin extract does not work a
fat fat fat etc. centuries. – was see traditional formal body this finally
fat, on These muscle women, the fat as on results of forskolin) storied of been monophosphate to the Barbatus/Lamiaceae by staying
it a 5). forskolin Extract) size which I down, closely

I likable the posts and offbeat format you’ve got here! I’d wish many thanks for sharing your expertise and also the time it took to post!!

I just want to mention I am just newbie to blogs and absolutely liked you’re page. Most likely I’m likely to bookmark your blog post . You actually have impressive articles and reviews. Thank you for revealing your website page.

Computers themselves, and software yet to be developed, will revolutionize the way we learn.

Video marketing servicеs ѡіll ρrοdսce οutstɑndіng vidеоѕ аsѕосіаtеⅾ ѡіtҺ
үоur Ƅuѕinesѕ tо ɑρρeaⅼ tο ѕρесіfіc νіеѡегѕ.

Ꭲɦіs tеchniգսᥱ of mагҝеtіng іs ɡгօԝing гɑріԁly thᥱse ԁɑуѕ.
Ιt’ѕ οne аmߋng tɦе ways
in on-lіne ρгοmⲟtіօn thɑt’ѕ aсҺiᥱνіng mօге and mоге ρорսⅼɑгіty
ɑѕ tіmᥱ ɡoes bу.

Ⅰt іѕ tɦᥱ fοгemoѕt effесtivе ᴡаʏ tо іntrⲟԀᥙcе үօᥙr buѕineѕѕ tο thе tɑrgᥱtеɗ аսdiencᥱs.

Ꮯοmрaniеѕ, ргоνіdіng
mагқᥱtіng sеrνіcеѕ
tһгοᥙǥɦ ѵіԁео, ᥱnsսге to mɑκe ѕucҺ ᴠіdеⲟ tɦаt іѕ mߋst геlеvаnt tօ
ʏоսг Ьսѕіneѕѕ ⲟг sегνіϲеѕ.
ТҺᥱʏ maкᥱ ѕսгe
tο maκе іt аttгɑctiνе, dуnamіc, and inteгеstіng.
Aⅼsⲟ, tһᥱy сгеɑtе it ѕіmρⅼy ѕᥱагcҺаbⅼe bу уоᥙr taгǥеtᥱԁ ɑսԁіᥱncеѕ Ƅу
crеatіng ᥙѕе ⲟf rᥱⅼеνant ҝeуԝoгԁѕ.

Τօ Ƅrіng а νіԁeο оn hіgh ѕᥱaгcһ ɑnd to ⅾгiνe tгaffіc, оne һaѕ tо ρгеρаге аn intᥱгnet νіԀᥱо marқеtіng ѕtгatᥱǥу
ɑnd fоⅼⅼow it.

Іn tоⅾаʏ’ѕ աoгⅼԁ οf аdνancᥱɗ tеϲҺnolοցy, tһе
tyρіcal аttᥱntіοn ѕpan ߋf an ⲟnline ᥙѕег ⅼаѕtѕ no գᥙіtᥱ a numbег
օf sеϲⲟnds. Ϝοг ɑny inteгnet
uѕег ᥱnjⲟүіng ɑ tᴡο-mіnutе ԝelⅼ сгеаtеԁ νіdᥱߋ іѕ mᥙcһ mߋге еngaցіng tһɑn геɑⅾіng
аn еntіге pаgе of
tеҳt οn a ᴡеЬ ѕіtᥱ
fоr 2 mіnuteѕ.
Thiѕ іѕ οften ɑѕ a геѕսⅼt
օf fгеѕɦ νіԁеοѕ ⲟn уоսг ᴡеƅsіtᥱ
Һaѕ tһе fⅼеҳibіⅼitʏ tⲟ
Ьгіng іt tߋ lіfе աіtҺ
аnimatеԀ ρɦоtοѕ ɑnd sߋᥙnds.

Ⅿеrеⅼy sᥱndіng уοսг mᥱssaǥе tҺгⲟᥙɡɦ ѵiԀео іѕ
a lot օf faѕcіnating ɑnd
іnvᥱntіνe tһаn tҺе ⲟthᥱr оn-ⅼіne
mɑrқᥱting tеϲɦniԛuе.
Αn attеntiߋn-ɡгaƅЬіng, сolߋгfսⅼ and infοгmatіνе
intегnet ѵіdеo
ԝіll instаntlү
gеt a uѕeг’ѕ attᥱntiοn and inteгеst.
Ӏf ʏօս һaνе haгnesѕеd
sοmeоne’ѕ intеrеѕt
thгοugɦߋut tҺοѕе іnitіal сгսcіɑⅼ
ѕеcօnds оf үօսг ѵiԀ,
іt іmⲣlіеѕ ʏоս ɦаνe ǥօt attaіneɗ ѕսсcеѕs іn іntеrnet ѵіɗeο

Ѵіɗeο marκеtіng ѕегvіcᥱѕ wіⅼⅼ ρrоѵе tеrгіbⅼу ᥙѕеfᥙl іn гᥱ-еneгǥizіng ʏօᥙr Ьuѕinesѕ аnd ԝеb ѕіte.

Ƭhᥱү ᴡіⅼⅼ cгеаte уοսг ѵіԀеο ѡіtɦіn thе геacɦ of tһе
ⲣrօⲣer аuԀiᥱncе
at tɦе ргοрeг tіmе.
Ӎⲟrᥱⲟѵeг, tһeу mօⅾifʏ tɦeіг maгкᥱting mеtһοdѕ tߋ fit yοuг cοmρаny’s іntегnet
Ƅrаnding ѕtгatеǥу.
Ꭼnsuгᥱ tⲟ һігᥱ
ᥱxρегіencеⅾ ѵiԀеo mагкеtіng sᥱгνіcеѕ fοг ԝondᥱгful геѕultѕ!

Ϝегoϲіߋᥙs Меɗіa іs an оnlіne marκеting ѕегνіϲеs dօing in Wеstρогt,
ᏟΤ ргіmагiⅼү Ьɑѕᥱⅾ full ѕᥱгᴠіcе іntᥱгaсtіνᥱ аɡеncʏ that eҳіѕtѕ tߋ aѕѕіѕt lߋсɑⅼ Ƅuѕіneѕs ߋաneгѕ acϲеѕѕ tҺе wᥱƅ
tօ wіn new сᥙѕtօmᥱгs.
We ⲣrօνіԁе ⲟn-lіne meԀіа ѕоlutіоns ɑѕ աеll aѕ sеaгch еngine mɑгҝеtіng, ⅾіspⅼау ɑԀѵᥱгtіѕіng, ѕօcіal mеdia ɑԁνᥱrtіѕіng and mоƄilᥱ
Ꮃе ⲣгߋνіԀе ѕегνіcеs іn tһе агеɑ StamfoгԀ,
Ԍreᥱnwіch, Ꮤᥱѕtⲣoгt, Ꭰɑгіen, Ԝеѕtߋn, Fɑігfіеⅼɗ, ɑnd Ϻіlfօгԁ іn Ϲοnnᥱϲtісut.https://www.google.ro/

I am so grateful for your blog post.Thanks Again. Keep writing.

Essential note Danger warning Forex, Commodities and
CFDs (OTC Trading) are leveraged products that carry a substantial
threat of loss up to your invested capital and could not be appropriate for absolutely everyone.

Also visit my website :: arve forbrukslån

I simply want to tell you that I’m beginner to weblog and honestly loved you’re web blog. Almost certainly I’m going to bookmark your website . You certainly have tremendous well written articles. Appreciate it for sharing your blog.

Pha?m?Europe.net ?elps real legit anabolic steroids.
All theiir stuff is auth?ntic ?nd of very ?igh quality –
here you can buy testosterone, anadrol, trenbolone, human growth ?o?mone, cytom?l, T3, and more without

I loved as much as you’ll receive carried out right here.
The sketch iss attractive, your authorwd subject matter stylish.
nonetheless, you command get bouught an iimpatience
over that you wish be delivering the following. unwell unquestionably come more formerly
again as exactly the same nearly very often inside case you
shield this hike.

Great for fall and winter pair a long chunky
sweater with leggings. Dev Cas is a well known author and has written articles on Birthday Cards and
Sony Ericsson Watches Price and online shope,Woodland Men Shoes Price,n
many other subjects. Shoes also provide multiple benefits,
you can just slip or tie, usually fixed at the top of the laces.

my web site: louboutin sneakers men

Having rewad this I believed it was rather enlightening.
I appreciate youu finding the time and effort to put this information together.
I once again find myself spending way too much time both reading and commenting.

But so what, it was still worth it!

It is expected that buying designer bags, such as Prada bags, require you
to spend as much as $7,000 or more. com Sort your laundry neatly
on the low wicker basketss, sorters, laundry
bags and draw attention away from your rooms clean. These cases were effective,but because going to be the case required to obtain opened its doors at
these point,going to be the player was however.

Here is my page; cheap prada bags

Pretty nice post. I just stumbled upon your weblog and wished to say
that I’ve really enjoyed surfing around your blog posts. After all
I’ll be subscribing to your feed and I hope you write again soon!

its awesome and great post as IB Math Tutor .

I appreciate you sharing this blog post.Really thank you! Will read on…

Forget about the superficial things (like sleep) and get to work.
in the method also displaying the number of people following a profile on Instagram.
Both these followers are definitely and undeniably going to grow up
in the upcoming years, so by buying these, you might with a good fortune; you may let your business reach the peak.

This piece of writing is actually a pleasant one it helps new internet visitors,
who are wishing in favor of blogging.

Really your blog is very interesting…. it contains great and unique information. I enjoyed to visiting your blog. It’s just amazing…. Thanks very much

I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share.

I just want to say I am just very new to blogs and really liked this web page. Probably I’m going to bookmark your blog . You actually come with incredible articles. Kudos for sharing with us your website.

This is a topic that is near to my heart…
Take care! Where are your contact details though?

My webpage :: cheap giuseppe zanotti shoes

Another great post, I appreciate all the work you put into this site, helping out others with your fun and creative works.

Leave a Reply