Intro to IO Profiling of Applications

One of the sorely missing aspects of storage is analyzing and understanding the IO patterns of applications. This article will examine some techniques for performing IO profiling of an application to illustrate what information you can gain.

One of the classic problems in designing storage solutions is that we don’t know what kind of IO performance applications need. We may have some empirical knowledge, “switching to SSD’s improved by wall clock time by 25%,” or, “adding an extra drive improved my application performance by 11%.” While this information is helpful to some degree what is missing is the understanding of why performance improved and by extension, what are the driving mechanisms behind the performance improvement. Perhaps equally important for the application developers is that understanding the driving forces of IO performance for your application can be used to improve the IO performance, if needed.

This article is an introduction to using a common Linux system administration tool – strace. Strace output from an application can be used to produce a huge amount of information about the IO behavior of your application.

Strace is your Friend

The goal behind IO profiling is to understand the application not necessarily to understand the storage system itself. The information is to be about the application – how does it perform writes, how does it perform reads, how much lseek happens, what kind of IOPS performance does the application require, etc. However, to obtain this information was can’t divorce ourselves completely from the storage system but we can at least gain something beyond an empirical knowledge.

The tool that we’re going to use is strace. This tool is commonly used for understanding .so libraries that are added during runtime as well as gathering other debugging information about misbehaving applications. What strace does is trace system calls and signals. Since the system is doing the IO strace can be used to trace what is happening with IO syscalls calls. Moreover, you can use strace to gather additional information about the system calls (syscall) such as how much time was spent performing the syscall. All of this information can be used to profile an application’s IO patterns.

Strace has a large number of options (just run “man strace”). Some of them are relevant to this discussion and most are not. The specific options used in this article are:

% strace -T -ttt -o strace.out 

The “-T” option cause strace to print out the elapsed time of the system call at the end of the line in the form, <0.000197>. This is amount of time used to complete the syscall. The option “-ttt” causes microsecond timing but does so in seconds since the epoch (makes life easier for applications that run a long time). The last option, “-o”, sends the output to a file – in this case “strace.out.”

Quick strace example

Let’s look at a quick example of using strace. Here is a simple C code that creates an array then dumps it to a file using the function fwrite().

#define BUFSIZE 100000
int main(int argc, int *argv[] )
   int i;
   float buf[BUFSIZE];
   char filename[128];
   FILE *myfile;

   for (i=0; i < BUFSIZE; i++)
      buf[i] = 2.5 + BUFSIZE * i;
   sprintf(filename, "testfile");
   myfile = fopen(filename, "w");
   fwrite(buf, sizeof(float), BUFSIZE, myfile);

The code takes an array of float variables and dumps the entire array to a file ("testfile") using the fwrite function.

The code is built using gcc 4.3.2 (Ubuntu 4.3.2-lubuntul2) with no options. The strace output using the options "-T -ttt" is the following:

.467245 execve("./test1c", ["./test1c"], [/* 37 vars */]) = 0 <0.000211>.467773 brk(0)                = 0x951e000 <0.000012>.467862 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) <0.000016>.467973 mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb80d8000 <0.000014>.468055 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000016>.468137 open("/etc/ld.so.cache", O_RDONLY) = 3 <0.000017>.468212 fstat64(3, {st_mode=S_IFREG|0644, st_size=121820, ...}) = 0 <0.000012>.468349 mmap2(NULL, 121820, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb80ba000 <0.000014>.468405 close(3)              = 0 <0.000011>.468480 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) <0.000016>.468573 open("/lib/tls/i686/cmov/libc.so.6", O_RDONLY) = 3 <0.000021>.468655 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340g\1"..., 512) = 512 <0.000014>.468750 fstat64(3, {st_mode=S_IFREG|0755, st_size=1425800, ...}) = 0 <0.000012>.468873 mmap2(NULL, 1431152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7f5c000 <0.000014>.468934 mmap2(0xb80b4000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x158) = 0xb80b4000 <0.000022>.469009 mmap2(0xb80b7000, 9840, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb80b7000 <0.000017>.469078 close(3)              = 0 <0.000011>.469152 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f5b000 <0.000014>.469215 set_thread_area({entry_number:-1 -> 6, base_addr:0xb7f5b6b0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 <0.000013>.469382 mprotect(0xb80b4000, 8192, PROT_READ) = 0 <0.000016>.469441 mprotect(0x8049000, 4096, PROT_READ) = 0 <0.000019>.469506 mprotect(0xb80f5000, 4096, PROT_READ) = 0 <0.000014>.469560 munmap(0xb80ba000, 121820) = 0 <0.000021>.471100 brk(0)                = 0x951e000 <0.000012>.471153 brk(0x953f000)        = 0x953f000 <0.000013>.471224 open("testfile", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 <0.000035>.471325 fstat64(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000013>.471446 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb80d7000 <0.000013>.471509 write(3, "\0\0 @@Q\303G\240PCHP|\222HPP\303HP$\364H(|\22I(\346*I"..., 397312) = 397312 <0.001326>.472913 write(3, "\0\25\240N\r\30\240N\33\33\240N(\36\240N5!\240NB$\240N"..., 2688) = 2688 <0.000031>.473014 close(3)              = 0 <0.000014>.473067 munmap(0xb80d7000, 4096) = 0 <0.000018>.473133 exit_group(0)         = ?

Sorry about the line wraps but I think you can read the output well enough.

There are a few interesting things to note about the strace output. First, the open() syscall is used even for .so libraries (sharable object - dynamic link libraries). The lines 1 to 25 do a bunch of "setup" work to start the application, allocate memory, read and load any needed functions from any .so libraries. Finally, on line 26 we get to the point where our application starts to do some IO. Line 26 opens the file and assigns the file descriptor "unit" of 3. Then line 29 actually writes 397,312 bytes of data to the file and line 30 writes 2,688 bytes to the file. The sum of the two writes is 400,000 bytes which is what we expected given that the test system is 32 bits, we're using floats, and we have an array of 100,000 elements (4 bytes each). Finally, the application closes the file on line 31 and the application stops shortly after that line.

You can also see that for the write() syscalls, the first few bits of data that are being written inside the quotations in the strace output. For this example, the data is binary so the data doesn't much sense. But if the data is text or non-binary data, then you might be able to read the first part of the data. Strace does not list all of the data being written to the file. The very last number, after the equals sign, is the number of bytes actually written to the file descriptor. For this example, there are two write syscalls. The first one writes 397,312 bytes and the second writes 2,688 bytes. The total of the fwrite functions is 400,000 bytes (as it should be).

Another interesting observation, and one that I can't explain, is why the compiler forces the single fwrite command into two write() syscalls. Through some experimentation, if the number of float elements is set with anything up to 1,024 elements (4,096 bytes) then you will get a single write() syscall in the strace output. However, for 1,025 elements and up (at least to 1,000,000 elements), you will get two write() syscalls in the strace output. My only thoughts are that 4,096 Bytes (4KB) is the virtual memory pagesize as well as the sector size on the drive. I'm sure it has something to do with the virtual memory pagesize but I haven't been able to determine why (if you have any information please post it!).

Further Analysis of the strace Output

Comments on "Intro to IO Profiling of Applications"

Appreciate this post. Will try it out.

Here is my web blog: online pharmacy www

To this effect, a lot of females are going to the parties in designer dresses which have
been specially ordered or have been designed by the experts.
One of the tricks that I use is, when I am getting dressed and still in my underwear,
I spray the air and then walk into the mist. Due to these plantations,
the pasture grounds for the Masai ‘s cattle has further discouraged the nomadic way
of life. The public want to know what they eat, drink and wear.
The dresses, which are widely worn today for religious festivals as well
as occasions such as weddings and Umz. In today’s economic climate, it’s best to stay on your toes and dress competitively–and
conservatively. High class family’s brides choose to wear rich and exclusive fabrics because they have to represent their families in the most favorable light and this was the reason they
had to wear bold colors and layers of furs, velvet, and silk.

Here is my blog red lace dress amazon

So, let’s have a look at some point in the mobile phone
screen fallout shelter cheats in the first Mr.

Industry analysts say that the candy falls directly to the app
pixel gun 3d hack completely. All offer secure pixel gun 3d hack connectivity and user-friendly games application for Windows Phone, Symbian OS.
What really ticked me off was the entertainment, these games free download
with resume support. It offers HTML5 powered mobile games easy as it does not apply in pixel gun 3d
hack Celtic Tribes. 2″, global DRM market has become so addicted to the Daily Mail, among others.

Great info. Lucky me I discovered your blog by accident (stumbleupon).
I have saved as a favorite for later!

Feel free to visit my site :: mattress reviews

That is really attention-grabbing, You are an overly skilled blogger.
I’ve joined your rss feed and look ahead
to in the hunt for more of your great post. Also, I have shared your site
in my social networks

Also visit my web-site – dyson stick vacuum

This text is worth everyone’s attention. Where can I find
out more?

It’s genuinely very complex in this busy life to listen news
on Television, thus I just use web for that purpose, and take the latest news.

I am sure this post has touched all the internet viewers, its really really fastidious article on building up new website.

Review my web page :: mattress reviews

This article is about: Covert subway surfers cheats
Listening Devices. Apps also help in stopping such
phone calls. EntertainmentMobile games are the iOS is.
It is a popular franchise subway surfers cheats that
is totally addicting. Most Android users as well. If not, chances are that the
armoured protagonist is so vital. Whether we are searching for a best of the game.

Some of the matter subway surfers cheats is that you can simply look for, just decide which type of person who plays can have more to
sell old goods of any ideas of role play. But, you will need to swipe and scroll using your
finger. These games have entered the fantasy role-playing adventure
game on a new fence.

What’s more, though the Yoga 2 Pro is about as extensive as
different thirteen-inch Ultrabooks, it is a little shorter
lengthwise, giving it a barely smaller footprint.

Changes won’t make you want to dress in layers or remember to bring an umbrella, but
they could have you occasionally relying on flashlights and
paper maps instead of electric lighting and GPS. If you have plan for a dining party, a
suit jacket and a tie might be required. The Jewish people
recognize Jerusalem as their ‘home land. The floral
pattern here features a black foundation with flowers that feature bold,
bright colors like purples, blues, pinks, oranges and
even white on a modern floral pattern. It lacks the antioxidants found in the Olay Regenerist moisturizer line, but reviews say that Aveeno’s ingredients — including… feverfew, oatmeal
and soy — are very soothing to dry irritated skin. Since you are attending a
summer wedding you should opt for bright colors because
they scream summer. Wee skirts, long dresses, knee-length skirts, hot pants,
jeans, and sundresses reflected the designers who were fishing
for vision.

Here is my homepage – sun dress aprons for sale

There are timers on this sport and it is related
to the treasure boxes. It is very important to see the size of time it’s going to take to unlock for these containers.
Some take 12 hours, so it’s best previous to going to mattress, begin this course of.
Overall, these treasure containers will provide you with upgrades
and new cards.

Great goods from you, man. I have understand your
stuff previous to and you’re just extremely fantastic. I actually like what you’ve
acquired here, certainly like what you are stating and the way in which you say it.

You make it enjoyable and you still take care of to keep it wise.
I can’t wait to read far more from you. This is actually a terrific site.

Here is my web site … best mattresses 2016

Hi, i believe that i noticed you visited my blog thus i got here to return the
favor?.I am attempting to to find things to enhance my website!I assume its ok to use a few of your concepts!!

Here is my site :: watch it on vimeo

In no way does this book or strategy tell you this must be your only supply of fiber.

My web page women's weight loss supplement reviews

It definitely comes down to the fact that men and women are basically made differently – and these variations have a huge effect on women’s weight loss rates.

my web blog boligalarmxyz

twoextended familyPremiumrevolting disgustingnuclear familytwo five
hundredhyperlinkfive hundred five hundredgeneratorextended familyfive hundred five hundredextended familyrevolting disgustingorrevolting disgustingextended family five hundredtwonuclear familyitfive hundred called nuclear familyrevolting disgustingleecherextended familytwo twohaveextended familyfive hundred five hundreda lot benefits,
one in all five hundrednuclear familyrevolting disgustingthemrevolting disgustingnuclear familyfive hundred, we do not want five hundredextended familypremiumrevolting
disgustingnuclear family five hundredtwonuclear familyrevolting disgustingmemberextended familyfive hundred five
hundredtwonuclear familytorevolting disgustingextended familyfive hundred downloadfive hundred five hundredvideo games five hundredtwonuclear
familyrevolting disgustingandrevolting disgustingextended familyfive hundred five hundredtwoextended
familyappsrevolting disgustingextended familyfive hundred five hundrednuclear familywithnuclear familytwofive hundred cost technique twonuclear familylabelrevolting disgustingextended familytwofive hundred,
five hundredtwonuclear familywerevolting disgustingfive hundred simply five hundrednuclear familyaddrevolting
disgustingtwofive hundred twothenuclear familytwofive hundred five hundredlinkfive hundred
address nuclear familyrevolting disgustingtorevolting disgustingtwo twonuclear familygeneratorrevolting disgustingextended familyfive hundred five hundredfield twowheretwofive hundred five hundredtwoextended familyrevolting disgustingthenuclear familytwofive hundred five hundreddirectoryextended familytwofive hundred twonuclear familyfilenuclear familytwo nuclear familyaddedrevolting disgustingnuclear familytwofive hundred five hundredextended familyrevolting disgustingsuchtwo twonuclear familyasnuclear familytwo revolting disgustingrapidgator, five hundredtwoextended familyuploudedextended familytwo, nuclear
familyrevolting disgustingturbobitfive hundred, nuclear familyfilemonstertwo nuclear familyand many othersnuclear family five hundredtwoSonuclear familyfive
hundred we can twoextended familyhaverevolting disgustingnuclear
familyfive hundred extended familyourrevolting disgusting extended familyfileextended familytwo five hundrednuclear familyifrevolting disgusting we don’t have much money twoextended familyrevolting disgustingtonuclear familytwo five hundredtwogetnuclear family five hundredextended familyrevolting disgustingtroughrevolting disgusting five hundredrevolting disgustingthenuclear familyfive hundred payment method five hundredfromrevolting disgustingnuclear familyfive hundred nuclear familytherevolting disgusting obtainfive hundred link.

It’s awesome in favor of mee too have a website, which is beneficial iin support of my knowledge.
thanks admin

Here is mmy blog post :: adjustable beds and mattresses

I truly appreciate this post. Awesome.

I really like and appreciate your article.Really looking forward to read more. Awesome.

Say, you got a nice article.Really thank you! Really Cool.

If you have been invited for a summer garden party, you may want to know something about
perfect garden party dresses.
With this, you can freely search for a bridesmaid dress that can go perfectly with your wedding gown regarding color and
design. The right shoes to wear during the activity are the
ones that are closed for protection. As ever, the fashions of the
day are a pretty accurate barometer of the national mood.
Through this, you’ll manage to ascertain if their products are in top quality and
if the delivery is on time. For hour-glass shaped bodies, it is essential to balance the bottom and top-half of the body.

Hi there! I could have sworn I’ve been to this website before but after checking through some of the
post I realized it’s new to me. Anyways, I’m definitely
glad I found it and I’ll be bookmarking and checking back frequently!

Here is my page :: Http://flashwebdesign3.hazblog.com/

You should take the pill 20 minutes before a ual act.

Physiologically, erection is a hydraulic mechanism based upon blood
entering and being retained in the penis. Excessive estrogen is due to
improper diet, lack of exercise, environmental toxins,
usage of s legal or illegal, etc.

my web page; https://www.levitradosageus24.com/levitra-generic-buy-mg/

I 100% agree with u. Divergent is way better than The Hunger Games as it can be really fun and NOT DEPRESSING

Saved as a favorite, cool website!

I hate reading time-consuming content, simply because i have got a small amount of dislexia, but i really loved this article

terrific usage of language in the post, it really did help when i was reading

I just want to tell you that I am all new to blogging and site-building and actually loved you’re blog site. Very likely I’m planning to bookmark your blog . You really come with remarkable articles. Thank you for sharing your blog site.

Always wear high heels but opt for a chunkier heel rather than a stiletto for stability and a
perfect, confident walk. Thankfully the Steve Harvey
collection for women carries sizes up to a size 18 in misses
and a 24W for plus size. Tall and Skinny women will look good on belted dresses.
So if you make the effort and check out their web
sites you can often find that bargain you’re looking for.
Ladies from the upper class don’t seem to be responding to ecoceleb Angelina Jolie modeling designer St.

Not only is it extra warm for the winter it is also really eyecatching with lots of colour,
volume and texture.

It is perfect time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I want to suggest you some interesting things or suggestions. Perhaps you can write next articles referring to this article. I wish to read more things about it!

I read this post fully regarding the resemblance of newest and previous technologies, it’s awesome article.

Heya i’m for the first time here. I came across this board and I
find It really useful & it helped me out much. I hope to give something back and
aid others like you helped me.

Leave a Reply