Chasing The Number

In the United States, applying to college usually means taking the SAT test (Standard Aptitude Test). The goal of course, other than taking a bunch of money from every kid that wants to go to college, is to provide a common number to compare students from around the country. Presumably it is a predictor of how well a student will do in their first-year of college. Not everyone agrees, however.

Trying to reduce a person’s ability to one number is convenient and at the same time naive. If it were that simple, why ask any other questions. What if your college or job application asked for your name, SAT score, and IQ. A simple binary decision. “But, but…”, you plead, “there is more to it than that.” Of course there is because once you are in college the SAT score is useless, it does not help you further your education in any way. How well you perform in college and beyond is up to you. The same can be said for your IQ test (if you have ever had one).

Many people tend to agree with my analysis and at the same time are enamored with the Top500 List. As I have said before, “the list” is very useful and provides a good historical record and “snapshot” or the HPC community. I believe problems arise when the list is used for that which it was not intended. Quite often you will read “the X fastest computer in the world”, where X is a spot on the list. To which I always add, “running a single benchmark that has little bearing on many HPC applications. Results are voluntarily submitted and many machines are not on the list.” Such, arguments usually go unnoticed when writing the press release about the “new University supercomputer that is the X fastest computer in the world.” Which is why I am glad to hear that chasing the Top500 is maybe less important than has been in the past. A recent article in the Chronicle of Higher Education seems to support this notion.

In my opinion, this change is probably coming from several directions. First and flippant, the analyst’s expect the Chinese to own the list in very short order. Of course, there may be some competition for the top spot, but buy and large, the Chinese are dumping a lot of money into HPC. Why focus on a race you are going to loose.

Second and probably more to the point, results help funding for future grants while a good Top500 position does not. In a sense the “Top500 bubble” may be bursting. I consider this a good thing because there are plenty of other HPC issues that need addressing. For example, taking my SAT analogy even further, does a good Top500 placement predict how well the cluster will perform in the first year of service? Certainly if all the users are running linear algebra codes, but how well does the Top500 reflect other applications. What about utilization. Does a good list rank indicate users will get enough work done on the cluster? Lastly, current budget pressures or “doing more with less,” certainly cannot be discounted as a reason to not even consider getting your machine ranked.

To be fair, there are other methods to measure HPC performance. The HPC Challenge (HPCC) is a set of seven tests (the Top500 benchmark included) that give a much better picture of the “overall” machine performance. The HPCC tests are similar to a report card rather than a single test result. Of course, the old HPC adage that “your applications are the best benchmark” is still true and any sizable cluster purchase should include this type of testing.

One growing HPC sector is high data throughput machines. These machines are designed to allow processing of large data sets and therefore depend on high I/O rates as much as number crunching capability. Indeed, some data heavy problems would bring your typical Top500 beast to crawl. The Top500 is of little help assessing these machines.

In closing, Top500 performance is important, as is the HPCC, and the Green500 for that matter. Measuring complex systems from many angles helps us build better devices. My wish is that some of the effort and energy that goes into trying to build a faster HPC sports car goes into allowing more people to travel on the highway.