In the book Dreaming in Code , the author mentions Axe Sharpening, specifically how development teams can spend too much time sharpening their axe and not enough time cutting down trees.
"Give a person six hours to cut down a tree, the saying goes, and she will spend the first four hours sharpening the axe. In other words, most of us would rather spend time improving the tools that make a job easier than getting on with the job itself".
However what if you axe becomes completely blunt?
The problem - Too much unwieldy code
The JIRA team uses JWebUnit and HttpUnit to help us do functional testing of JIRA.
The current JIRA web test framework design is based on a class called JIRAWebTest, which is derived from WebTestCaseWrapper, which is derived from WebTestCase, which is itself a delegate/mixin for the WebTester class, and finally is derived from the JUnit TestCase class.
JIRAWebTest is 7300 lines long (is this a bad code metric?), while the WebTestCaseWrapper is a thin little thing at 1100 lines long.
And just for good measure, we have approximately 230 actual "Test" classes all in the one package. Just to keep things interesting :)
These tests consist of approximately 80,000 lines of code, which has grown from 20,000 lines of code in 2 years. And because we are trying to get bettter at testing, it will continue to grow. According to Clover, our combined unit test plus functional test coverage is 72% and its growing at 0.1% per fortnight.
What else is bad about it?
None of the code above is unit tested, which may explain why assertTextSequence() has gone through several iterations, each time causing passing tests to fail and failing tests to pass.
So, will we hit the magic 10,000 lines of code in one class? Can we top 500 classes in one package?
What does the functional test framework do?
The code consists of a bunch of methods that perform the following functional areas.
- General Assertions
- Text Assertions
- Table Assertions
- Link Assertions
- Page Assertions
- JIRA general administration operations
- JIRA navigation operations
- JIRA data setup operations
- Database restore for test setup
- Logging and HTML failure dump processing
How could this happen?
Everything seems perfectly obvious in hindsight.
- Obviously the size of the base classes are too big.
- Obviously the problem domain concerns of the code is too concentrated.
- Obviously the code needs refactoring?
- Obviously packages should be used to organize the classes.
What happened is that the functional test framework code is a little "unloved". The functional test code grew over time but we didn't take the time to ensure that the framework code was robust, refactored and re-usable.
Because of the large number of methods in a main classes, new team members have trouble "getting to know" the framework and what methods they can re-use. And hence we end up with multiple methods doing almost the same thing.
We didnt spend enough time sharpening our "functional testing" axe. And now we are beginning to pay for it in. Its proving harder to re-use the functional test code, its proving hard to improve the functional test code and its proving hard for the team to know about the features the functional test code offers.
Whats good about it?
Even with its current unwieldy code design, its still works. It runs 1-20 times a day in Bamboo and runs every night in our nightly build.
The use of web functional tests is all about "insurance" for the future. It has saved our bacon on a numerous times where we have regressed JIRA behavior as new features are added or when we have fixed other bugs. It helps enormously to catch these problems during development rather than after release.
What are we going to do about this?
The big question is how should we proceed in the future without destroying the work of the past? We cant afford to refactor the current code but we don't want to continue "growing" the functional test code based on this framework.
The answer is to create a new functional test framework that can exist along side the old framework. We keep the old tests and begin writing new functional tests using the new framework. This minimize the impact of the change and also ensure JIRA continues to be "tested" according to the existing "rules".
An alternative would be to go back and refactor the old test code to use the new framework. The risk is that you cant "faithfully reproduce" the exact testing rules and hence the "testability" (is there such a word?) of JIRA goes down. And we don't want to take that risk.
Where to from now?
We are currently in the early stage of implementing the new web functional test framework. We hope to do the following things:
- Have web test classes derive from a class as close as possible to JUnit TestCase.
- Break out the assertions from the helper methods and rely instead on the base JUnit Assertions directly.
- Re-organize the code into meaningful packages
- Provide "helpers" that do things like "HTML DOM inspection", HTML Table inspection, HTML link inspection
- Also provide more "high level helpers" that do JIRA administration and navigation operations.
The goal is to then write all new tests using the new framework and to slowly "retire" the old code. It will still exist and be run to test JIRA but it wont be actively changed.
By following a few more coding best practices (god I hate that term), in essence give the functional test code a bit more "love", we hope that it will be better used by development team members, more maintainable over time and able to cope with our expected test growth.
3 Comment(s)
I'm not sure I agree that your only solution is to "start over" with new tests. I'm speaking from a bit of my own experience here and have gone through the same scenerio you mention and it just leads to a maintenance nightmare. For example, when a UI or HTML screen changes and you "break the build", now you have to update *two* tests instead of one.
The existing tests could be refactored *if* you can ensure you don't reduce coverage. You should be able to do this by configuring clover to fail the build if test coverage drops after making changes to your test classes. Sure, it might be a bit slow to ensure you keep up the test coverage, but it's better than letting the current code "rot" and have to re-write all of the tests with a new framework.
By Ryan Sonnek at January 22, 2008 6:36 PM
Ryan,
I think elements if what you say is true and it would be more ideal to migrate the old tests to any new improved way. And yes we will have to edit 2 places if we change the HTML dramatically and/or the "structure" of the information on a page.
But we have ~ 80,000 lines of "old test code". Its not broken per se just badly organised and hard to extend and maintain. We considered migration and discounted it because the benefits (consistent code / one place to edit) are far outweighed by the risks (losing test meaning / porting mistakes / huge reverse engineering cost)
For example say we change something and 2 old tests break . We have to go and reverse engineer the test "meaning", fix it for the new change. Lets say it costs 2 hours per test. So that is 4 hrs cost incurred per incident.
Whereas if we go through the 80,000 lines of code, we are guaranteeing us a cost of 320 test classes * say 20 methods each * 30 minutes each method = 8000 hours of effort.
We dont reckon we are going to break enough tests over time to make it worth while to backport the old test code.
Now it might not actually costs that much because you know how estimates blow out when you do simple multiplication but you get the idea.
By Brad Baker at January 22, 2008 8:17 PM
I'm curious why you're using JWebUnit in this circumstance, especially since you're already testing through the full stack (including the HTML presentation layer). Why not raise the level of abstraction with WebTest, which is really just an Ant-based wrapper for HTMLUnit? You can always get right back down to the HTMLUnit API if you want, but for most functional acceptance tests, WebTest is all you need to drive and verify your web interface. HTMLUnit (and hence WebTest) already provides robust helper methods (""helpers" that do things like "HTML DOM inspection", HTML Table inspection, HTML link inspection.") It's pretty trivial to quickly verify tables and links with XPath expressions.
If you don't like the idea of using XML-based tests (coders can find them a little verbose), you can always use the Groovy Builder support, and write the tests in Groovy.
BTW, I have nothing against JWebUnit, but for an app as complex and configurable as JIRA, I can see how 80,000 lines of JWebUnit could really paint you into a corner.
By Nate at February 16, 2008 7:31 PM