Performance Blog

Posts Tagged ‘faban

I’ve been working for awhile now to revamp the entire stats processing and graphing design in Faban. For those who haven’t heard of Faban, it is a performance tool that helps creation and running of workloads. Faban currently uses a technology called Fenxi  to process and graph stats. Fenxi has given us lots of problems over time – it is poorly designed, lacks flexibility and doesn’t even seem to be maintained anymore. So I decided to get rid of it entirely.

I am really excited by the changes. I think this is one of the major enhancements to Faban since Akara (the original Faban architect) and I left Sun/Oracle.

So without much further adieu, here are the upcoming changes:

  •  New cooler looking dynamic graphs

I’m using jqPlot that produces nice looking graphs by default. The ability to zoom in on a section of the graph is really very very nice, in addition to actually seeing the datapoint values as you mouse over the graph. This one feature I think will make the Faban UI  more modern.

  •  Provide support for graphing stats from Linux.
   The Fenxi model does not support multiple OS’s well. So I’ve got rid of it completely. Instead, support for Linux tools (I currently have vmstat, iostat in) is added by using the post-processing functionality already baked into faban. The post-processing script will produce a ‘xan’ file (named after Xanadu, the original internal name for Fenxi). The nice thing about the xan format is that it is highly readable by a human. Take a look at any of your current detail.xan files produced by Faban. Very easy to read, so I’m sticking with this format.
  • New Viewer to graph all xan files
Of course, the above 2 enhancements are not possible without a way to actually interpret the xan files and convert them to jqplot json format. So a new Viewer has been implemented that renders the xan file nicely – both tables and graphs.
I’m attaching a screen-shot of a sample Linux Vmstat output to whet your appetite for the new features.
Stay tuned. I hope to check everything in the next couple of weeks.
If you are a Faban user, please join the Faban User google group.
Tags:

As many of you know, Faban is a free and open source benchmark development and automation framework. It was originally developed at Sun Microsystems, Inc. which made it available to the community under the CDDL license.

With the architect and lead developer Akara Sucharitakul and myself no longer working at Oracle (not to mention the demise of the project website without notice), we decided to host it at http://www.faban.org. The website isn’t pretty but at least it hosts all the documentation,  a downloadable kit and a pointer to the source on github.  In the coming weeks, I will work on organizing the site. A big thanks to all the folks who expressed concern about the future of this project. With your help, we can continue to support it.

If you are a faban user, please do join the new Faban Users forum at http://groups.google.com/group/faban-users.

 

Tags:

After finishing my post on the Faban 1.0 announcement, I realized that it was geared towards users who were already using Faban. So I decided to write this post for users who have never used Faban.

Faban is two things :

  1. A framework for developing performance and load tests.
  2. A tool to run these tests and view their results.

The former is called the “Faban Driver Framework” and the latter is called the “Faban Harness”. Although the two are related, it is entirely possible to run an arbitrary test developed outside of Faban using the Faban Harness. In fact, many benchmarks do just that. In this respect, Faban is rather unique.

The Driver Framework

The real power of Faban is unleashed only when you use the framework. The framework provides capabilities similar to other load testing tools, namely : emulate users, keep track of response times and other metrics, run monitoring tools etc. Some unique features of Faban include the ability to :

  • accurately measure server response times at the network layer
  • emulate a Markov model to realistically model web workloads
  • emulate a Poisson process for inter-arrival times
  • support hundreds of thousands of users with the lowest possible resource requirement

If that doesn’t convince you to try Faban, maybe some of the features in the Harness will. Please read on.

The Harness

The Faban Harness is a web application that queues and executes runs, displays reports and graphs from previous runs and in general serves to maintain the results repository. Some features of the harness include :

  • Gathering of configuration information across all the systems in the test configuration (including driver systems)
  • Automatic collection of system level performance monitoring data
  • Ability to run arbitrary scripts/commands to collect any type of monitoring data
  • Automated management of a wide variety of common server applications like apache, MySQL, glassfish etc.
  • Graphing of both workload and monitoring data for easier analysis

If a lot of these features sound like LoadRunner and other fancy, high-priced tools, they are (in fact these tools don’t even have all of the functionality I’ve listed above). And you get all this for free in an open-source tool !

Tutorials

So check it out. The easiest way to get started with Faban is using the Quick Start Tutorial. It gives step by step instructions on installing Faban and running a pre-built workload including screen-shots.

If you are ready to start creating your first workload, check out the Creating your first Workload Tutorial. For users new to Java, this step by step tutorial should make it really use to get started.

Faban 1.0 has just been released. This is a major milestone for this open-source workload creation and test framework. Faban is widely used by many performance geeks to performance test various types of server applications. Amongst open source tools, Faban is unique in that it not only provides a framework to create and run performance/load tests, but also has robust functionality to run monitoring tools. It collects all sorts of configuration information as well to truly help performance engineers keep track of configuration and tuning settings.

Here are some major new features in 1.0 which I think will make Faban a very attractive proposition (when compared to the likes of some very expensive proprietary tools).

Automated Form Generation

If you have used Faban before to create workloads, this is the feature you probably have been craving. But let me explain for those who have never used Faban.

Unlike most other tools, Faban does not provide a standard GUI to drive workloads. Instead, a workload developer needs to create a custom form for each workload using XForms. This provides great flexibility and power in designing workloads as you can tailor the information you collect to the workload. On the other hand, editing XForms is a nightmare and error-prone and most benchmark developers spend the most amount of time in this one task. No longer. Faban 1.0 auto-generates the XForms for you from your run configuration file (run.xml) at deploy time. How cool is that !  And the generated forms are hard to tell apart from good hand-crafted ones – they are that good !

Dynamic Services and Tools

Faban always had the concept of Services (meaning objects that were responsible for managing a particular server application like Oracle, Apache etc.) A Service is used to start/stop/restart a server, grab it’s configuration and error logs etc. But to add a new service, one had to add a class to the Faban code. This meant a user had to not only understand Faban code but also had to get the new service code committed. The same was true for monitoring tools (For example, I added ‘mcstat’ – a tool for monitoring memcached but this had to go in as part of the Faban harness code).

With 1.0, both services and tools can be dynamically deployed. This means you can write your own services and tools and deploy them in your own environment. What’s more – Faban comes with several sample services to get you started. These include services for Apache, Glassfish, Memcached, MySQL, Oracle etc. I have added services for Thin and Nginx as part of Olio using this framework. See the documentation for more details.

Tags and Feeds

As the number of tests and results grow, we have all faced the situation where we can’t find a particular run result. Tags can greatly help here. Faban 1.0 has  web2.0 features including tags and RSS feeds. You can add tags when defining a new test (actually it’s easy enough to add them later as well, so you can go back and tag all your old runs if you want to). To find a particular test (or series of tests), you can do a Tag Search from the main Faban page.

You may wonder what in the world is the use of a Feed in a performance tool ? Well, if you kick off a series of runs before you leave work, you can sign up for the RSS Feed (located at the top right corner of the screen). This will send you an email whenever a run completes. So even if you don’t have access to your test systems, you can view the results anywhere – perhaps on your iPhone while you are at dinner ?

This post is already too long. I’ll cover the other new features in a future post. But meanwhile, do check out Faban 1.0.

I mentioned fhb in a prior post, when it was still in it’s infancy. Both faban and fhb have now come a long way and so I wanted to post an update.

Faban is both a framework for creating workloads and a harness for running them while providing performance monitoring of all parts of the system being tested. There are myriads of free and open source load testing tools out there but none that match the power, functionality and accuracy of faban (My own opinion of course, but I think you will agree once you evaluate it yourself).

To make use of the flexibility and power that faban provides, you do need to write some code in Java. As soon as that sentence is uttered, I can see some heads shaking and worse, it sends shivers down some spines ! For some reason, some people seem to think that writing code is bad, or difficult. And writing in a programming language like Java – what could be worse ! I don’t want to get into the merits of programming/scripting languages etc. but the fact is that the kind of programming you have to do for faban is largely declarative with some high-level method calls. Faban uses annotations heavily, provides the build scripts and does all the heavy lifting for you – here is some sample code from the ‘web101’ sample benchmark :

@BenchmarkOperation (
name    = “MyOperation1”,
max90th = 2,
timing  = Timing.AUTO
)
public void doMyOperation1() throws IOException {
logger.finest(“Accessing ” + url1);
http.fetchURL(url1);
if (ctx.isTxSteadyState())
contentStats.sumContentSize[ctx.getOperationId()] +=              http.getContentSize();
}

That’s it. You now have a web request that will do a GET on url1 and keep track of the size of the response data.

But for those who think this is too hard, faban provides the full functionality (almost !) of the driver framework in a simple command-line tool called ‘fhb’ (for Faban Http Bench).  Countless benchmarks have now been run using ‘fhb’ and it is the only tool that gives you the power of modeling transitions in Markov chains using a simple XML configuration file. See the tutorial for how to use fhb.

With a few ‘fhb’ runs under your belt, it should make the transition to the full-fledged Faban Driver framework much easier so I hope you will give it a try. Doing quality performance testing is hard work and a good tool is essential if you want to get accurate results.

Did I mention that faban and fhb are free and open source ? And actively supported via a user forum ?


Pages

Latest Tweets

Categories

Archives