At Atlassian we believe in eating your own dogfood. We've therefore setup Bamboo to continuously build and integrate our own products.
Over time we keep adding new types of builds (plugins, tools, etc) and existing builds start taking more time (number of testcases increases, more CPU intensive testcases, etc). We therefore decided to setup a new Bamboo box dedicated to running all the JIRA automated tests on a number of different JDKs and app servers. Some of the stats from this new box were pretty amazing and worth sharing here.
On our old build box all we could really afford was to run one functional test build of JIRA for the Enterprise edition. This build usually took around 70-100 mins to complete. Pretty slow really and we were seriously considering cleaning up our TestHarness to make it run faster.
On the new box, all functional tests run in about 20 mins . It's a pretty late model quad core Xeon. Even with 3-4 functional builds running in parallel the build times didn't increase much more than 1-2 minutes for each build. Pretty impressive. All of JIRA compiles in about 10-15 seconds. The Unit tests take about a minute to run in total.
One could argue that spending 3+ days trying to clean up the JIRA TestHarness such that it performs better is a complete waste of time, as we can simply buy a faster box again in 2-3 years time. Question is I guess what costs more: A new box or 3+ days of development with no visible benefit at all for customers.
The other bit that I found interesting was the performance increases between JDK 1.4, JDK 1.5 and JDK 1.6:
- JIRA Functional Tests for Enterprise JDK 1.4: 25 mins
- JIRA Functional Tests for Enterprise JDK 1.5: 19 mins
- JIRA Functional Tests for Enterprise JDK 1.6: 16 mins
Note: For JIRA we have 2 automated test suites, Unit tests (about 2600 of them) and Functional tests (about 430). Functional tests startup an instance of JIRA and make HTTP requests against this instance testing various parts of the UI.
6 Comment(s)
Hi Andreas,
Which test tools are you using for unit and functional tests?
Thanks.
By Mustafa Tan at January 8, 2007 10:21 PM
One could argue that spending 3+ days trying to clean up the JIRA TestHarness such that it performs better is a complete waste of time, as we can simply buy a faster box again in 2-3 years time. Question is I guess what costs more: A new box or 3+ days of development with no visible benefit at all for customers.
Wouldn't you risk removing important test cases by cleaning up your TestHarness, possibly impacting the quality of your product? If so, I'm happy you went for the extra box. I wonder if you could compare the cost in customer satisfaction vs. cost of the new box.
By Chris at January 9, 2007 4:10 PM
Hi Mustafa,
For unit tests we simply use JUnit. The functional tests are a little more complicated. For setting up the app-server we use cargo. To actually run the tests (i.e. clicking around the UI) we use jWebunit.
All the tests (both unit and functional) get executed via Maven. Bamboo simply runs the appropriate maven command.
Cheers,
Andreas
By Andreas Knecht at January 9, 2007 4:51 PM
Hi Chris,
We probably wouldn't remove testcases from our harness, but identify testcases that perform badly and try to clean these up. You are right however, there is the risk that changing testcases will result in less accurate/incomplete tests.
Cheers,
Andreas
By Andreas Knecht at January 9, 2007 4:55 PM
Re: is spending time on improving the unit tests a waste of time.
I always find that what you don't understand has a habit of biting you in the bum and in this case, contributing to the general cruftyness of your test cases. If you continually maintain your tests and harness then it is unlikely to arrive at a point in the future (or will arrive at it more slowly) where is needs a serious time consuming rework.
Re: Risking removing test cases (Chris)
I also find fear of change to actually be a paralysing force, and a much too easily available justification for procrastination. If the brittleness of your test or harness, or the confidence in the quality of your own work is such that you are actively trying to avoid such change, then you have bigger issues; but ones that you needs to address and work through rather than avoid...
By George Janczuk at February 8, 2007 4:49 PM
Agreed! (and hello btw. Didn't expect a message from you on here ;)) With the JIRA test harness, performance isn't a huge problem (yet). We are continually improving and adding test-cases (both functional and unit), so it is in a pretty good state. Having said that there are some areas for improvement and we'll probably look at them when our test performance starts to degrade significantly on the new build box as well.
By Andreas Knecht at February 8, 2007 5:16 PM