Archive for the ‘Workloads’ Category
Olio was developed by Sun Microsystems as a way to compare, measure and analyze the performance of various web2.0 technology stacks. We had a great collaboration with the RADLab in UC Berkeley and gave this project to Apache. However, with the take-over by Oracle, Sun was no longer willing to support the project. Many users continued to find and use Olio but no one (including big-name companies like VMWare who used it for their own benchmark) was willing to contribute to it. I’ve always felt that open source works only when there is big corporation support, but I digress.
Anyway, I’ve asked for the Apache Olio project to be wound down. For those who may still be interested in using it, I have now copied over the repository to github – feel free to fork it. I have also moved some of the documentation to the wiki.
For anyone considering moving a svn repository to git, git-svn was mostly painless. It preserves the history of the edits which is really great.
Olio is a web2.0 application that can be used for performance testing of various web infrastructure technologies. The cool thing about Olio is that the same application (a social calendar similar to yahoo.com/upcoming) is implemented in PHP, Rails and Java. Thus it can be used for comparison of different stacks (to answer questions like “Should I move my PHP app to Rails? “) or compare different software infrstructure components (apache vs nginx say) or compare different hardware. Olio is currently an apache incubator project. You can get an overview of Olio from this presentation.
Release 0.2 was a lot of work. I was the release manager for both Olio releases done so far. 0.2 went a lot more smoothly (anyone who has done an Apache release knows all the criteria and rules involved !)
The major features of this release are :
- First release of the Java version. The Java code has been in the svn repository for some time now and has been actively used by VMWare and others. Understandably, it was fairly buggy and a lot of fixes went into creating this release.
- Re-organization of the file store. Previously, the entire unstructured file store (all the photos and documents) were in a single directory. This caused severe navigation problems at loads of several thousand concurrent users as the number of files grew very large. The file store now has an heirarchical directory structure.
- The load generator (driver) has been upgraded to use HttpClient 3.0.1. This eliminates some of the complexity in the driver when dealing with multi-part posts, re-directs etc.
- The cycle times for the AddEvent and AddPerson operations have been adjusted to result in passing runs in most cases. Previously, the minimum cycle time was defined as 0 which caused problems when the 90% response time was several seconds. The cycle time curve would keep shifting to the right making it impossible to meet the requirements. This is a topic for a separate article, so more on it later.
- Olio simulates browser caches being cleared when new users first login by re-fetching all static files 40% of the time. When the cache was alive, it did not issue any such requests. It turns out that for older apps that do not use the Cache-control directive, the browser does issue a small request for every static file with the If-modified-since header. To better reflect these sites, Olio 0.2 now issues these if-modified-since requests. This results in substantially more traffic to the server.
Because of these major changes, any performance results obtained using Olio 0.1 should not be compared with those from Olio 0.2.
If you are interested in a realistic web workload to test your LAMP stack, Rails stack or Java EE stack, please do consider downloading Olio.
After finishing my post on the Faban 1.0 announcement, I realized that it was geared towards users who were already using Faban. So I decided to write this post for users who have never used Faban.
Faban is two things :
- A framework for developing performance and load tests.
- A tool to run these tests and view their results.
The former is called the “Faban Driver Framework” and the latter is called the “Faban Harness”. Although the two are related, it is entirely possible to run an arbitrary test developed outside of Faban using the Faban Harness. In fact, many benchmarks do just that. In this respect, Faban is rather unique.
The Driver Framework
The real power of Faban is unleashed only when you use the framework. The framework provides capabilities similar to other load testing tools, namely : emulate users, keep track of response times and other metrics, run monitoring tools etc. Some unique features of Faban include the ability to :
- accurately measure server response times at the network layer
- emulate a Markov model to realistically model web workloads
- emulate a Poisson process for inter-arrival times
- support hundreds of thousands of users with the lowest possible resource requirement
If that doesn’t convince you to try Faban, maybe some of the features in the Harness will. Please read on.
The Faban Harness is a web application that queues and executes runs, displays reports and graphs from previous runs and in general serves to maintain the results repository. Some features of the harness include :
- Gathering of configuration information across all the systems in the test configuration (including driver systems)
- Automatic collection of system level performance monitoring data
- Ability to run arbitrary scripts/commands to collect any type of monitoring data
- Automated management of a wide variety of common server applications like apache, MySQL, glassfish etc.
- Graphing of both workload and monitoring data for easier analysis
If a lot of these features sound like LoadRunner and other fancy, high-priced tools, they are (in fact these tools don’t even have all of the functionality I’ve listed above). And you get all this for free in an open-source tool !
So check it out. The easiest way to get started with Faban is using the Quick Start Tutorial. It gives step by step instructions on installing Faban and running a pre-built workload including screen-shots.
If you are ready to start creating your first workload, check out the Creating your first Workload Tutorial. For users new to Java, this step by step tutorial should make it really use to get started.
Faban 1.0 has just been released. This is a major milestone for this open-source workload creation and test framework. Faban is widely used by many performance geeks to performance test various types of server applications. Amongst open source tools, Faban is unique in that it not only provides a framework to create and run performance/load tests, but also has robust functionality to run monitoring tools. It collects all sorts of configuration information as well to truly help performance engineers keep track of configuration and tuning settings.
Here are some major new features in 1.0 which I think will make Faban a very attractive proposition (when compared to the likes of some very expensive proprietary tools).
Automated Form Generation
If you have used Faban before to create workloads, this is the feature you probably have been craving. But let me explain for those who have never used Faban.
Unlike most other tools, Faban does not provide a standard GUI to drive workloads. Instead, a workload developer needs to create a custom form for each workload using XForms. This provides great flexibility and power in designing workloads as you can tailor the information you collect to the workload. On the other hand, editing XForms is a nightmare and error-prone and most benchmark developers spend the most amount of time in this one task. No longer. Faban 1.0 auto-generates the XForms for you from your run configuration file (run.xml) at deploy time. How cool is that ! And the generated forms are hard to tell apart from good hand-crafted ones – they are that good !
Dynamic Services and Tools
Faban always had the concept of Services (meaning objects that were responsible for managing a particular server application like Oracle, Apache etc.) A Service is used to start/stop/restart a server, grab it’s configuration and error logs etc. But to add a new service, one had to add a class to the Faban code. This meant a user had to not only understand Faban code but also had to get the new service code committed. The same was true for monitoring tools (For example, I added ‘mcstat’ – a tool for monitoring memcached but this had to go in as part of the Faban harness code).
With 1.0, both services and tools can be dynamically deployed. This means you can write your own services and tools and deploy them in your own environment. What’s more – Faban comes with several sample services to get you started. These include services for Apache, Glassfish, Memcached, MySQL, Oracle etc. I have added services for Thin and Nginx as part of Olio using this framework. See the documentation for more details.
Tags and Feeds
As the number of tests and results grow, we have all faced the situation where we can’t find a particular run result. Tags can greatly help here. Faban 1.0 has web2.0 features including tags and RSS feeds. You can add tags when defining a new test (actually it’s easy enough to add them later as well, so you can go back and tag all your old runs if you want to). To find a particular test (or series of tests), you can do a Tag Search from the main Faban page.
You may wonder what in the world is the use of a Feed in a performance tool ? Well, if you kick off a series of runs before you leave work, you can sign up for the RSS Feed (located at the top right corner of the screen). This will send you an email whenever a run completes. So even if you don’t have access to your test systems, you can view the results anywhere – perhaps on your iPhone while you are at dinner ?
This post is already too long. I’ll cover the other new features in a future post. But meanwhile, do check out Faban 1.0.
I have been working on trying to get the next release of Apache Olio (the web2.0 workload) out for some time now. It’s been delayed due to lack of resources. Although it is a single project, it really is a three-in-one deal as there are three separate implementations of the application in Olio.
The applications are non-trivial and as such trying to update them to ensure they are in sync isn’t an easy job. Not only do we have apps implemented using different technology stacks but also the corresponding load drivers. The driver part is easier – since we use Faban and it’s all Java (and it’s 0.2 ready). But the apps require knowledge of Rails, Java EE and PHP and currently I’m stumped by the lack of rails resources.
Despite what is said about ruby and rails, I find rails code to be difficult to comprehend. I should perhaps invest some time in learning it, but right now it would be great if I could find some kind soul with Rails expertise to help out. The critical issues that need to be fixed are well isolated and I can point you to exactly where in the code the changes need to be made (so you don’t have to understand the entire Olio app). If you’re interested in contributing to an apache project, please do respond and thank you in advance.
The first cut of a Java EE implementation of Olio is now checked into the repository. The file docs/java_setup.html gives instructions on how to build and setup this implementation. The implementation uses JSP, servlets, JPA for persistence, yahoo and Jmaki widgets for AJAX etc. The web application is located in webapp/java/trunk and the load driver, database and file loaders etc. are in workload/java/trunk.
Check it out.
We have just released the first binary version of Apache Olio for both the PHP and Rails implementation. Both implementations have been tested quite thoroughly now and we think they are robust enough for serious use – especially for performance testing the workloads.
I introduced Olio in a previous post. It is a toolkit that includes a sample web2.0 application implemented in both PHP and Rails that includes a load generator to drive load against the application.
Please visit the Olio site and download the kits. If you find it interesting, I invite you to come join the project.