If you are using scrum and agile within your organization, you already know about the daily stand-up meeting and the value its brings to the team. Many organisations who have not fully adopted scrum still find the stand-up meeting to be extremely useful when done properly.
Pyxis Technologies have just released a really neat iPhone/iPod Touch app - the _agilely Timer. The application allows you to efficiently facilitate daily stand ups, roundtable discussions and manage simple timeboxes. It costs $1.99 with all monies raised going to FIAN - FoodFirst Information and Action Network.
As Francois Beauregard said to me, "start standing-up for something important".
At Atlassian we believe in "agile" development (similarity to a very popular buzzword recently unintended). We strive to be as agile as possible and clearly see the benefits.
Sure, there are teams who are "more agile", there are teams who are less. Almost every team applies different agile practices. That's OK as we believe in self-organisation and bottom-up evolution. Still we all believe in the 4 fundamental principles of agile philosophy, which are compatible with our company values.
When Crucible became part of the Atlassian suite, a lot of us wondered how code review would match our agile environment. Two years later, code review has become an integral part of our agile developers' day-to-day jobs. Some say they are addicted to it. Let me tell you why.
I am not going to elaborate on an obvious advantage of code review: improved code quality. Instead, I would say that code quality is a nice side effect, but not the purpose of code review in our teams.
Code review is one tangible way that we enact the first principle of the Agile Manifesto: "Individuals and interactions over processes and tools". Here are some examples of how code review helps us collaborate better:
New team members
Atlassian is constantly growing. We hire passionate, often quite- experienced developers who need to dive into our non-trivial code base very quickly. In addition to new hires, existing team members sometimes rotate to different temporary or permanent positions.
Code review makes ramping up on a new code base faster and less painful, and helps maintain engineering knowledge across the team and the company over time. People more experienced in the code base review newbies' commits (there's no better way to learn the product than to start coding bugs or stories), and then advise them on which design inconsistencies they introduced and which shared components or utilities they should use and how. In return, the newbies can bring invaluable experience, good engineering practices, useful libraries, nice patterns, tricks, etc. from their former teams, resulting in fresh insights to improve the code base. In short, code review is a two-way conversation. Both the author and reviewers communicate and learn from each other.
Code review is even more important when a really junior developer joins a team. Senior developer don't have time to hand-hold the junior folks, since they need to be coding. That said, doing small but frequent code reviews of the junior developer's code is no big deal. The result? Senior developers can focus on writing code for most of the day and spend a 10-15 minutes a couple times per day to review the code changes by junior developers. The junior developers benefit from feedback, and the senior developers haven't lost valuable coding time.
Geographical distribution
Atlassian develops software on 3 continents, in time zones which differ by as much as 10 hours. Most teams working on a given code base are co-located (it really helps), but sometimes it's just not practical or feasible. Thus we ended up with a few geographically distributed teams - a big problem for some agile practices such as pair programming and daily stand ups.
Tools-supported code review comes to the rescue here by encouraging and facilitating knowledge flow between different locations.
With asynchronous code reviews, time zone differences can actually be an advantage. We often raise a review in the afternoon (as a good summary of our day's work), ask for review by colleagues across the ocean and go home. When we arrive at work the next morning, we have their review feedback: they were working on it while we were asleep! First thing in the day, we can answer questions and fix identified problems, and then proceed with our daily development tasks.
Integration
Code bases do not live in isolation. Most of Atlassian's products integrate with each other in some way or another.
For example, I work on IDE Connectors, which need to talk to Atlassian server products via remote APIs. These APIs are continuously evolving, often in response to the needs of the IDE Connector team. The server product teams are mostly located in Sydney, and their work day is ending just as ours is beginning.
Code review is the main way for the IDE Connector team to provide feedback about the remote APIs we rely on. Whenever there is a change related to a remote API, I am one of the reviewers - thus I can learn what is going on on the server side, constructively criticise the evolution of the API, and ensure that my teams interests are at least heard if not fulfilled.
A similar situation arises when you need to program something in the product you normally don't work on. I call it "guest programming" - e.g. you need to add some missing API or plugin point or just fix a problem which is only reproducible in your environment.
Code review enables guest programming by increasing courage of both sides. The team which owns the product is less reluctant to let an outside person touch their code base, and the guest programmer has more courage because other folks will review his code to prevent any nasty bugs.
Talking about courage - some of us actually feel anxious when deprived of code review. Unit tests establish a safety net which helps you be courageous to mercilessly refactor and improve the code while avoiding nasty regressions. Similarly, code review is another safety net, which gives you this feeling that your crazy ideas will be evaluated by others before they will go to production, and that your great design, utilities or tricks have a chance to be spotted and reused in other places by other team member. This is a real teamwork!
Code review may be difficult and costly to implement and sustain. Not every team or team member wants to do pair programming, and code review meetings can be too time-consuming. Obviously we use Crucible, which handles the logistics of a review and lets developers concentrate on the most valuable part of code review: thinking about the code and providing feedback.
In the next instalment I'll share a few pieces of advice and precautions related to implementing code review in an agile team or company as a standard process.
In the third and last instalment I would like to compare pair programming with code review and reveal a few rules around code review that evolved at Atlassian.
Have an opinion or question? Drop us a comment below!
"Lesson" photo courtesy of eddidit / CC BY 2.0
"Obelisk" photo courtesy of Kevin Dooley / CC BY 2.0
When I think of Agile, I tend to think of the awesomeness it can bring me as a performance engineer. I see short cycles where the code is usually always stable and features are delivered incrementally. I get to watch the performance of the system evolve over time. Unfortunately, this means I'm testing software all the time. Thats fine though - none of this is by hand - this kind of repetitive task is perfect for automation.
It's clear that automating testing is important - automated tests mean you have more time to do investigation while you make your build system do the repetitive tasks, and when your build system does the heavy lifting then it's easier to test more regularly. More regular testing means you find problems sooner and go home on time.
In this blog, I'm going to go over how to take your JMeter performance tests and put them into a Maven build which you can run on your continuous integration server as often as is needed. I'll introduce you to the Chronos Maven plugin which can control JMeter in a repeatable way and has reporting features which can be used to gain useful information about the performance test.
Prerequisites:
For this blog, you'll need the following prerequisites:
A working knowledge of Maven
Working JMeter Performance tests
A Maven repository set up for retrieving artifacts and deploying new ones.
Atlassian has a well maintained Maven infrastructure - if you're just getting started, I'd suggest you read Sherali Karimov's great guide on setting up Maven infrastructure here.
The JMeter tests used in this paper are fairly complete, however this is not necessary for your first pass at getting the automation running. Its more important to get started and then improve what you have as your need more features.
When you are writing your JMeter tests, I would suggest keeping the information about the data you're going to be testing separate from the testing logic. By this, I mean keep the "flow" of your test in JMeter but use CSV files to define what URLs or options need to be used when testing. For example, instead of hardcoding a list of pages to sample rather put this into the CSV and load that into JMeter using the "CSV Dataset" controller. This way, you can change the CSV files to test a different set of pages or data in your application.
Here are the main steps I'm going to cover:
Create a performance test Maven artefact
Write a Maven pom.xml for the performance tests
Package & deploy the tests to the Maven repository
Write a Maven pom.xml to run the performance tests against an application
Run the tests
Automate the tests with Bamboo
George Barnett discusses performance testing inside Atlassian
Create a performance test artefact
In order to use your performance tests in an automated environment, you need to get them into Maven. Below are the components for a sample artefact, starting with the directory structure to use.
For the public-distribution.xml which contains information about how to package the artefact
jmeter
JMeter performance tests
resources
CSV resources files
In this test artefact, the CSV resource files which are distributed should contain the bare minimum information needed to run the test - the actual data you will use will be stored in another project. If your application has a demonstration dataset then it would be good to add information about this data to the CSV files as an example. Don't leave the files blank - it makes it easier for others if you have a simple structure inside as an example.
To deploy this project using Maven you'll need to write the pom.xml. I've attached a sample pom.xml to this post but here's some of the things you need to take care of:
Atlassian's performance tests are public. For this reason, I include the Atlassian Public pom as a parent. This ensures that this project inherits the correct settings and will be deployed to the correct place.
To keep things easy, set the version of the pom to the same as whatever software you're testing. When the software you're testing releases, bump the version to follow them (don't forget to tag the previous release!).
Maven assembly plugin. This uses the public-distribution.xml configuration file to package the tests and the sample dataset into a .zip and .tar.gz archive. It specifies which files and directories should be included into the generated archive.
Maven clean plugin. This plugin is set up to remove any .jtl files from the results directory to prevent leaking any test results into the archive file. .jtl is the extension given to output files that Listeners save result data in during the JMeter test run.
Packaging the JMeter test
If everything is in place you should now be able to use Maven to package and deploy the JMeter tests to your Maven repository. This is the first step to being able to include them in a repeatable test.
First, check that the artefact generated is correct:
spoem:trunk gbarnett$ mvn verify
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Performance Tests
[INFO] task-segment: [verify]
[INFO] ------------------------------------------------------------------------
[INFO] [site:attach-descriptor]
[INFO] [assembly:attached {execution: public}]
[INFO] Reading assembly descriptor: /Users/gbarnett/fisheye/performance-test/trunk/src/main/assembly/public-distribution.xml
[INFO] Building zip: /Users/gbarnett/fisheye/performance-test/trunk/target/performance-test-2.0-SNAPSHOT.zip
[INFO] Building tar : /Users/gbarnett/fisheye/performance-test/trunk/target/performance-test-2.0-SNAPSHOT.tar.gz
...
Check the resulting ZIP files to ensure that they're packaged correctly. Check that all the files you need are there. For an example of what a working artefact looks like, have a look at the FishEye performance test artefacts here
Once you're happy with this, you can deploy to the remote repository.
spoem:trunk gbarnett$ mvn deploy
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Performance Tests
[INFO] task-segment: [deploy]
[INFO] ------------------------------------------------------------------------
[INFO] [site:attach-descriptor]
[INFO] [assembly:attached {execution: public}]
[INFO] Reading assembly descriptor: /Users/gbarnett/fisheye/performance-test/trunk/src/main/assembly/public-distribution.xml
[INFO] Building zip: /Users/gbarnett/fisheye/performance-test/trunk/target/performance-test-2.0-SNAPSHOT.zip
[INFO] Building tar : /Users/gbarnett/fisheye/performance-test/trunk/target/performance-test-2.0-SNAPSHOT.tar.gz
[INFO] [install:install]
..
[INFO] [deploy:deploy]
..
[INFO] Retrieving previous build number from atlassian-m2-snapshot-repository
Uploading: https://maven.atlassian.com/public-snapshot/com/atlassian/performance/fisheye/performance-test/2.0-SNAPSHOT/performance-test-2.0-20080929.031627-1.zip
[INFO] Retrieving previous build number from atlassian-m2-snapshot-repository
Uploading: https://maven.atlassian.com/public-snapshot/com/atlassian/performance/fisheye/performance-test/2.0-SNAPSHOT/performance-test-2.0-20080929.031627-1.tar.gz
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41 seconds
[INFO] Final Memory: 14M/762M
[INFO] ------------------------------------------------------------------------
If you see this it means you have successfully deployed your JMeter tests to your Maven repository. If you find that Maven has failed then go over the setup carefully to check everything is in place. If you require more debugging information from Maven, you can run the above commands with the -X switch to enable more logging.
Once the build is successful and your artefacts are deployed to your Maven repository, they can then be included in your automated performance test. In the next part, I'll create project which will run the performance tests against your application. This Maven build can then be automated.
Directory structure for the automated test
The structure for the automated test is slightly different as you can see below. Create a new project in your SCM and create the directories listed.
The main difference is that we're using the test directory instead of main. This is because the main purpose of this project is to run tests instead of build an artefact.
It's best to start out by working out what actions (and in which order) you will need to take to test your application. In this example, here are the steps the test will perform:
Prepare the FishEye data directory using the chosen data to load
Prepare the Jprofiler config
Download and extract the FishEye application
Start FishEye
Run a JMeter test using the defined CSV data
Stop FishEye
Run reporting on the JMeter outputs and generate a site directory for upload
It's important to define what order this work should occur in before starting so you know what you are aiming for. I'm not going to annotate the whole pom.xml since it is several hundred lines long. I'll go over the important parts which you may not have seen before - I would suggest opening a copy locally before proceeding.
POM: Define build profiles
There will be many tasks to perform and settings to choose when running the test. Some of these tasks will need to happen every time (for example, setting up a database) and others will only need to happen when you are testing something specific (such as adding a code profiler). Maven provides the concept of Profiles (not to be confused with code profiling) which can be used to add and remove plugins to your test harness.
In this example, I will set up two profiles. By default, the information contained in them will not be used but when we do want to use it, it can be added to the test with a simple switch.
Profile
Task
profile
Run JProfiler before starting the build to gather CPU snapshots
dataset-sanity1
Controls the backend data to be loaded into Fisheye and the data CSVs to be passed to JMeter
The JMeter performance test artefact we packaged up earlier is a dependency in this test. Using the maven-dependency-plugin this artefact will be extracted into the target/test-classes/ directory for use.
The maven-dependency-plugin is also used to download and extract the application that's going to be tested.
The Maven exec plugin is used to start FishEye using the exec:exec goal. FishEye is started at the pre-integration-test phase so it's ready for use with the JMeter goals which run at integration-test phase. If your application runs inside an application server such as Tomcat or Jetty, you should use the Maven Cargo plugin to start and stop your application.
POM: Plugins - JMeter configuration
The JMeter test is run by the Maven Chronos plugin which is responsible for running JMeter. There a few settings which you need to pay attention to:
Where JMeter is installed but the base directory - JMeter should be in bin/jmeter under this directory. It's worth nothing that the plugin does not start jmeter via the startup script but rather invokes the jar's directly.
historydir
This variable should be a unique directory for the build. Since you're going to eventually have several profiles with different datasets and JVM versions, it should take this into account. I use the following format:
It's important to give JMeter enough memory. This is especially true if your test uses multiple Listeners. As you will see later when I cover reporting, the default report will include a graph showing Garbage Collection. This graph is actually JMeters garbage collection, but it's good to keep an eye on it. If JMeter spends too much time doing GC then the test results will be skewed and useless.
The settings in the attached pom.xml include some tweaks to the Garbage Collection, new size and survivor ratio.
POM: Plugins - JMeter System Properties
These variables are initially set as global configuration items but are then overridden in the profile for the dataset used. The variables here will be passed to JMeter using the -Dvariable.name=value command line switch.
JMeter accepts external variable input using the P function, eg: ${__P(script.runtime,1800)}
Using sysproperties in the pom, it's possible to override the variables inside the JMeter test. The most useful reason for this doing is to change the location of the resources which contains the CSV files which define which pages to sample directory. In the above example, the dataset.location variable is modified by the chosen profile in the pom and this is passed to JMeter, thus allowing the override of the resource directory used for testing.
The sysproperties shown earlier will be applied to all exections of the plugin. There are however two config variables above must be unique for each execution however:
dataid
This provides a unique name the test data output will be known as. It will be used to save files in the reports later and the savehistory goal will use this to save the data for this execution.
input
This is the .jmx performance test script to run.
Chronos Build Goals
Goal
Action
JMeter
Performs the test using the .jmx file specified in each execution
Check
Parses the output logfile and checks the samples against a ruleset in the pom.xml (not shown) to decide whether to fail the build or not.
Savehistory
Saves the history of the build to the history directory in order to build up long term data.
The reporting lifecycle produces graphs, one of which I've shown as an example at the top of the page. You should include one reportSet for each execution that was defined in the build part of the Chronos plugin. The dataid of the reportSet should match the relevant test execution dataid.
POM: Profiles
In order to control the dataset, I've added a profile for each piece of data I will be loading into FishEye. At the moment, I have only one, but making another will be easy with profiles. Most of the profile will be setting up variables which will be used elsewhere in the pom. For example, the dataset.location variable will be passed into JMeter using a sysproperty to set the resource base directory.
Once your pom is complete, you can run the test using the mvn command. Don't forget to have any profiles you need as part of the command:
$ mvn verify site -P dataset-sanity1
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building FishEye Performance Test Runner
[INFO] task-segment: [clean, site]
[INFO] ------------------------------------------------------------------------
[INFO] Preparing chronos:report
[INFO] [dependency:unpack {execution: 30_unpack-fisheye-zip}]..
[INFO] [exec:exec {execution: 60_start-fisheye}]
..
[INFO] [chronos:jmeter {execution: 71_fixedload-test}]
[INFO] Excuting test /Users/gbarnett/fisheye/automated-perftest/trunk/target/performanceTest/jmeter-test-fixedload.jmx
[INFO] Created the tree successfully using /Users/gbarnett/fisheye/automated-perftest/trunk/target/performanceTest/jmeter-test-fixedload.jmx
[INFO] Starting the test @ Fri Oct 03 12:32:53 EST 2008 (1223001173145)
[INFO] Display Summary Results During Run + 783 in 184.3s = 4.2/s Avg: 22 Min: 1 Max: 1377 Err: 0 (0.00%)
[INFO] Display Summary Results During Run + 2616 in 180.0s = 14.5/s Avg: 23 Min: 1 Max: 565 Err: 0 (0.00%)
..
[INFO] Display Summary Results During Run + 448 in 25.6s = 17.5/s Avg: 41 Min: 1 Max: 861 Err: 0 (0.00%)
[INFO] Display Summary Results During Run = 29095 in 1829.9s = 15.9/s Avg: 29 Min: 1 Max: 1377 Err: 0 (0.00%)
[INFO] Tidying up ... @ Fri Oct 03 13:03:25 EST 2008 (1223003005682)
[INFO] ... end of run
..
[INFO] [exec:exec {execution: 85_stop-fisheye}]
[INFO] [chronos:savehistory {execution: 71_fixedload-test}]
[INFO] [chronos:check {execution: 71_fixedload-test}]
..
[INFO] Generating "jmeter-fixedload-report" report.
[INFO] tests: 10
[INFO] jmeter samples: 29095
[INFO] generating charts.....
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31 minutes 39 seconds
[INFO] Finished at: Fri Oct 03 13:04:01 EST 2008
[INFO] Final Memory: 55M/746M
[INFO] ------------------------------------------------------------------------
The test has now run successfully. In the target directory you'll find a site directory which contains HTML and images to make up a build report. This will be uploaded to the URL specified in the distributionManagement section of the pom.
Automate the tests using Bamboo
Bamboo is a continuous integration server made by Atlassian and is perfect for running the tests regularly.
For information on creating a new build plan in Bamboo, see the document Creating a build plan in the online documentation for Bamboo.
In closing..
With this information, you can start to make a real difference to your development cycle. You'll find performance regressions faster and be on you way to code bliss. If you're interested in some of the processes which you can use to better integrate this cycle, see Andrew Prentice's blog on improving the QA cycle in You don't have to be Agile to be agile.
Learn more about software testing in an agile development environment
Part III: Technical Writing Insurgency (Part I | Part II)
"Reading the JIRA"
The Atlassian Technical Writers aren't clairvoyants, neither can we use The Force, but those would be really handy abilities in the documentation profession. The next best thing to supernatural powers of prediction is a technique we call "Reading the JIRA". Similar to reading the future in tea leaves, Tarot cards or the creases in your palm, reading the JIRA gives you a sense of things to come and critically, how plans and tasks are changing in the present time. Let me tell you, in an Agile project they'll be twisting and contorting like some mythical serpent in an old, violent legend.
We basically read every change that's going on in Atlassian's issue-tracking program, JIRA. As our own software developers are using JIRA itself to track their work which we are in turn documenting, by reading all of this JIRA material we have a powerful "early-warning" in place, that sometimes makes our prescience seem a little spooky. Milliseconds after a developer updates a spec in JIRA, we're immediately on the instant messenger, asking "How will this affect the documentation?". Sometimes they're a little shocked, responding with "Whoa! Easy there!". This may sound extreme, or perilously close to obsessive behaviour. Still, we've persistently found that by tracking the information flow continuously, monitoring changes Jedi style, we win critical days, hours and minutes that we then have up our sleeves to finish the job in time, no matter what's happened. The flip side of our obsession is that usually, none of the software developers would have thought to inform us about some critical change that turns our priorities upside down. They're bumbling along, hacking away in their IDE, humming blithely to themselves as the crisis bears down over days and weeks. Rather than a scheduled disaster, "Abracadabra"! We've magically got time to finish our work for the release.
At times, reading the JIRA is a lot like tapping the stream-of-consciousness from a high-school programming class packed with testosterone-fuelled egomaniacs, but that's the territory you step into when trying to probe the minds of Top-Gun software developers. Atlassian is most definitely full of those. It's competitive, intellectually vibrant and subject to tidal forces of opinion.
Authoring in Confluence
Another reason we are able to cope with the dynamic documentation work at Atlassian is due to our product, Confluence (which we also dogfood). Confluence is a Wiki, basically like a web server with pages that anyone can edit. For a Technical Writer, where our communication goal is purely informative, Confluence is a powerful solution. With established pages that we've released to the public, we can learn about a problem, find the page, edit the content, have it reviewed and re-publish it to the world in a small number of quick steps. We can have urgent maintenance work on the documentation completed in seconds, when all the pieces fall into place. With the inherent fast-editing capability in Confluence allowing anyone in the company to edit the pages, my subject matter experts are brought much closer to the content. It only takes a few minutes of their time and they can make all necessary changes immediately. This adds obvious bonuses to technical accuracy and speed of review (enabling multiple, iterative reviews to take place). Of course, professional writers are still essential to bring refinement, structure and enterprise standards of communication to the material.
If you're a Technical Writer about to embark upon an Agile adventure, you're in for an exciting ride. Throw out the rulebook, publish in a Wiki and get your developers using a quality issue tracker.
Finally, I will leave you with the words of Governor Bibble. Remember: "A disruption in communications can mean only one thing... invasion!"
Ed Dawson is the Technical Writing Team Leader at Atlassian.
Part II: Planning for Agile Documentation (Part I | Part III)
On Velocity
When a Technical Writer tries to resist the Agile process, the time you spend senselessly complaining and arguing your case could be better used acting decisively. In Part I, I discussed the dangers of resisting The Agile Way. Like surviving in a tropical jungle, to an extent you must give yourself over to the environment, rather than trying to fight it. This is not to say that there isn't a number of genuine Agile syndromes that make writers nervous.
One problem that you will see very early on is the issue of estimating work accurately in Agile projects. The difficult element is that user stories of equal size don't have equal documentation requirements. Remember, all the story-point estimation is being done more or less for the benefit of the hero in the Agile Universe, the software programmer. Some stories are reliably small, with small-ish documentation requirements (which is nice). Of course the converse is also true - some of them have an enormous communication payload (with a deceptively small story point count). Estimating stories and documentation impact is troublesome because as any Technical Writer will tell you, it really depends on the final polish of the feature. Thoroughly buffed and polished features generally require the least documentation. On the other hand, those raw prototypes and brand-new, first-generation solutions are the ones that really need a lot of supporting words. In my experience, the Agile process largely delivers features on the "raw" side of the equation - this leads me to a rather controversial statement: Agile Requires More Documentation (weather permitting). You may have a magical Agile team that delivers you wonderfully polished, iPod-like features, where you can say "Simply choose your music with the intuitive Click-Wheel!", but I'd personally expect that to be the exception rather than the norm.
Only once it is possible to lay eyes on a feature and see it demonstrated can the writer truly start to gauge the likely magnitude of the blocks of work. This is where the Agile term 'velocity' really takes on a life of its own. With Agile, these blocks come hurtling down the chute at a hundred miles an hour, rapid fire, in random order, ricocheting all over your work schedule. The day after the last sprint, the product is released to customers (with coding probably continuing throughout the day before). Naturally, this creates the need for an occasional 'tap-dance' to get the work done in time. As a Technical Writer, it sometimes seems as if the developers are all wielding pistols in some kind of Programmer's Wild West Saloon, chanting "Dance! Dance! Dance!" as their muzzles spew forth a fusillade of innovation and inspired code.
I paint a stressful picture, but Agile Tech Writing is also a positive experience in many ways. At Atlassian, we are nearly always writing documentation for 100% complete, brand new features. This is a wonderful contrast to some waterfall environments where you spend months trying to write instructions for vapourware, from vague business planning documents and specifications. Generally, Agile also delivers a fresh stream of new innovations which is a great alternative to tired re-hashes of an elderly feature set.
If this were a company that made mediocre products, the "dog-fooding" experience (where you must personally use the products that you make) would be torturous. Yet, because this is Atlassian, the very products we're dogfooding play a key part in sustaining our sanity.
Ed Dawson is the Technical Writing Team Leader at Atlassian.
In Part III, Technical Writing Insurgency, we'll cover the business advantages of being a clairvoyant, how to read a programmer's mind and leveraging JIRA and Confluence for documentation.
When you first begin Technical Writing in an Agile development environment, you face a number of immediate problems. The first is that you seemingly have far less time to do your work. With the developers choosing accessible goals and shipping them regularly, as a writer you feel that you're rushing. Sprinting (if you will), to use the Agile nomenclature. It's like stepping into a near-vertical mountain stream during the spring melt. It's bracing, turbulent, and full of razor-edged boulders.
Each milestone contains a number of user stories (and their component parts, story points) which the developers are churning out like mincemeat. If you don't catch the respective developers while they're working on these little pieces, they've moved on and can often be quite disinclined to shift their focus to the ancient past of last week.
I'd argue that one part of the problem is that Agile philosophies are essentially hostile to all non-programming parts of the delivery process. Where you once had Analysis, Design, Implementation, Testing and Documentation, you now have Planning Poker and Implementation Sprints. Analysis and Design have been savagely pruned. Testing (in its traditional Release Testing form) has been all but eliminated, due to Agile's reliance on integrated unit testing and peer review. Finally, the feeble proponents of Documentation, (always outsiders in the world of development) are easily shunted out the door by a philosophy so savage it has devoured parts of itself in order to become leaner.
Many a Tech Writer's response to Agile Development gives me flashbacks to Star Wars Episode 1: The Phantom Menace, specifically a character called Governor Bibble. A learned, middle-aged fellow, he was Queen Amidala's faithful council advisor. Bibble remains in the Imperial Palace during a full-scale invasion - while the Queen escapes to the forest to make battle plans. The rules of his world have just been turned upside-down.
Soon, the fearsome agents of the Trade Federation arrive, flanked by their terrible battle-droids. Bibble waits in the Imperial Court, still hoping for reason to prevail.
A Technical Writer's protestations against the Agile process usually run something like this:
Agile Nute Gunray (riding in a fearsome mechanical chair): "When are you going to give up this pointless strike? Your Queen is lost, your people are starving, and you, Governor, are going to die, much sooner than your people, I'm afraid."
Governor Bibble, Technical Writer: "We're a democracy! The people have decided... they will not live under your tyranny!"
Agile Nute Gunray: "Take him... away!" (Waves his arm dismissively. Droids march Bibble off to prison.)
The key with Agile Documentation is not to become Governor Bibble, but rather be more like the Queen. There is value in trying to secure sanity agreements on minimum levels of documentation and sensible time frames, but be prepared for documentation to be bundled as a "concurrent activity" (this seems to be something of an occupational hazard in Technical Writing). The time you spend senselessly complaining and arguing your case could likely be better used by acting decisively and switching to an insurgent strategy (which I discuss in Part III of this series). Waste no time - embrace the chaos!
Ed Dawson is Technical Writing Team Leader at Atlassian.
In Part II, Planning for Agile Documentation, we'll cover the puzzle of estimating Agile projects, the virtues of dog food and the Programmer's Wild West Saloon.
Test teams don't usually choose their organisations software development methodology. Hopefully the irony of being in such a situation is not lost on any teams that call themselves "Quality Assurance". Still this doesn't prevent test managers and test teams from adopting practices outside of the decreed methodology to their advantage. In this post I'll explain how one Agile practice, continuous integration, can benefit and be implemented by test teams working in non-Agile environments.
If you're testing in large organisations, following non-Agile development methodologies, co-ordinating large manual testing efforts, especially at places where making software is not the core business - I know your pain. I spent many years working in similar environments. The requirements are never accurate nor comprehensive (unfortunately they're not concise either, thanks to the 20+ page template full of meaningless boilerplate), the design keeps changing (see it just changed again), of course there is no contingency in the project plan to accommodate these changes (other than you working weekends) so the specs no longer reflect reality (don't worry you're working off the wrong version anyway) and the extra preparation time so graciously provided by development when they missed their deadline (naturally) evaporated in a scramble to accommodate a raft of late changes in scope (social networking...in a data warehouse?) as well as fend off attempts by the development team to sneak poor code into test ("It works fine on our machines!"). At least the stakeholders are staying true to their word by refusing to slip the release date.
Thankfully you're able to test more than you planned, in less time than was agreed, without the information and resources you were led to expect, because the knowledge that the advertising is paid for, the on-line viral campaigns have started, the marketing collateral is printed, the training has been delivered and the CEO is en route to the press conference keeps you from sleeping at night.
Some testers like to be the hero, working long hours to save the day. That's misguided, because real heroes don't take a hiding and then keep coming back for more. They win the fight and ride off with the prize. So if you find yourself in situations anything like that above, understand that your biggest fight is not with buggy code, poor requirements and bad specs. It isn't with project managers, developers, business analysts, architects or customers either. It's with the development process that ensures that test teams will always pay for the sins of the previous phases AND makes the customer suffer. How? Because longer days and working weekends can only make up some of the debt. Adding extra staff can cost as much as you gain. Eventually corners get cut. You know what I'm talking about. Tests important enough to think up, write down and schedule get descoped, major bugs get re-classified as minor, acceptance/exit criteria become meaningless and bugs get shipped that shouldn't have. Who's the hero now?
Don't do the best job in a bad situation, first improve the situation.
If you manage or perform functional testing there's a good chance your test plan describes the situation like this:
Regression testing is usually performed last because,
Bugs are more likely to be found in new untested code, rather than unchanged and previously tested code, so testing of new functionality should be prioritized first.
Bug fixes may themselves cause regressions, so to avoid repeating regression tests, regression testing is scheduled after bug fixes are complete.
This sounds like a commonsensical approach and it would be if this reflected reality. Alas the real situation looks more like this:
Invariably development finishes late overall, but usually some features are completed on time. This often tempts project managers and test managers to split testing into phases accordingly. Testers switching between phases will cost productivity later, but hopefully not outweigh the benefit of starting some testing as early as possible. Once phase 1 is complete, test teams might be tempted to start regression testing those functional areas not impacted by phase 2. Often times this has to be abandoned when late changes are added, because testers are needed to test the new changes and/or the changes invalidates much of the regression testing done so far. These late changes may also include changes to new functionality that's already been tested, so this needs to be retested as well before regression testing of the entire release can begin. Still this doesn't mean critical regressions aren't found once regression testing finally gets under way, requiring more fixes and another round of regression testing to be done. Rinse and repeat if you have more phases and more rounds of late changes. In the end testing takes much longer than planned, which may force short cuts to be taken (descoped tests, bug severities lowered, etc.), test managers get worn out by the constant reprioritizing and testers get fatigued from the constant task switching. But it doesn't have to be this way. Here's how...
Step 1: Automate your regression tests
For four reasons:
It increases the amount of time testers have to test new functionality. (That's manual exploratory testing. Which requires intelligence, creativity and adaptability. Which requires a human being.)
It increases the level of confidence in the regression test results (Let's be honest, humans are poor regression testers. Checking the same thing over and over is tedious. Unlike computers, we get bored, we get distracted, we don't notice regressions slipping past.)
Once automated you can run them any time you want (No more worries resourcing regression testing, just press "start".)
It increases testers job satisfaction - less context switching, no repetitive testing, more time to find bugs, greater use of mental abilities.
But aren't automated tests prone to breaking and costly to maintain? They are if they're poorly designed and testers don't have the skills to fix them. Record and playback test automation tools are easy to use but that's because the tests they create have no design. They're not designed to be efficient, flexible or robust, so it should be no surprise when such tests break and break often. It is possible to record a test with these tools and build the design in afterwards, but that's a bit like building a house first and renovating it to fit your design later. Sure you can knock out a wall here and there, but eventually the structural supports and plumbing are going to get in the way and you'll be forced to settle with an unsatisfactory mess. However, automated tests that are written with good design in mind don't break unnecessarily, because they're designed not to. Writing tests this way requires technical skill, which your team may not have. In which case you'll need automation specialists to get started, but you don't want to be reliant upon a few people to automate tests in the long term. Developing the entire test team so that every member has the skill to write well designed automated tests should be your goal. It means more automated tests can be written, in less time and allows the cost of maintaining and fixing automated tests to be shared across the team.
There's another reason for developing these skills across the entire team. Developer productivity is rapidly accelerating. They have smarter development environments, better tools, more and more third party libraries to use and more efficient development processes. They can produce more in less time. Without the ability to automate well, testers won't be able to keep up.
There's no shortage of automated test tool providers including many open-source options (www.opensourcetesting.org provides a comprehensive list). Different tools suit different contexts, so what we use at Atlassian is not necessarily right for you, but if you test Java based web applications then you should check out the free open-source tools that we rely on: JUnit, JWebUnit, HtmlUnit , HttpUnit & Selenium
Step 2: Make your automated regression tests fast
How long does it take to run your regression tests? I once worked at a company where manual regression testing took ten testers three weeks to complete. Obviously any regressions we found incurred huge costs in terms of delays and rework. At Atlassian our products aren't as large or complex as that system, but neither do our regression tests take three weeks to run. Depending on the product they take between half an hour and three hours to complete. That's up to 2000 tests (excluding unit tests of course). Imagine finding every single show stopping regression bug two hours after being given a release, not two hours before it's due to be released.
With fast automated regression tests, the previous situation starts looking less time-consuming and complicated:
There are many ways to make your automated regression tests fast, too many and too context dependant to cover here, but always keep the following in mind:
Don't test via the GUI unless you are checking the GUI. Rendering screens and web pages takes a lot of time, testing via an API or a headless browser will be dramatically faster.
Divide automated regression tests into sets that can be run in parallel e.g. we have a different set of tests for each web browser allowing us to test them all simultaneously, not sequentially.
Don't run tests that don't test what's changed. Clover's test optimization feature tracks the code coverage of each test, allowing us to automatically run only those tests which executed code that's changed since the last regression test.
Step 3: Use a Continuous Integration server to run your automated tests
So now that you have your automated regression tests and they run fast, why wait until the end of the test phase to run them? Now you can run them any time. All the time. Fully regression test every release, patch and fix immediately. Ah, but who's going to start the tests, collect the results, prepare the reports and publish them? If regression tests are being run every day, that's a lot of administrative work. Unless of course you automate that as well, which you can with continuous integration (CI).
CI is the Agile development practice of continually integrating code changes into the code base, building the software from that code base and testing it. It benefits developers by giving them feedback quickly on the changes that they've made. CI can be performed manually, but it's more efficient when automated and the tools that do this are known as continuous integration servers. Now compiling code and building software might not be of interest to test teams but the ability of CI servers to automate the execution of automated tests is. There are many commercial and open source CI servers available. At Atlassian we make and sell our own, Bamboo, and here are some of the things it can do:
Automatically store all regression test results in a central database
Bamboo is a test managers ultimate assistant. It won't be late for work, get sick or go on holiday so your automated regression tests always get run when they're supposed to and it'll take care of collecting and reporting your regression test results and status. With all the administrative workload taken care of then, there's no excuse not to run your regression tests all the time, so now our situation looks like:
To learn more about Bamboo you can download a free, fully functional 30-day trial version of this popular continuous integration tool. Alternatively take a look at the public Bamboo instance we provide for the Atlassian developer network.
Step 4: Integrate your automated tests into the build process
Once you have a continuous integration server running your automated regression tests, you also have the framework necessary for developers to continually run their builds. There's a strong case to convince developers to do this; the infrastructure is already in place and doing so will deliver these benefits:
Regression tests are automatically run during the development phase not just in test. "All regression tests pass" is now an entry criteria to the testing phase not an exit criteria.
Smoke tests need only test new functionality, given existing functionality has already been tested thoroughly.
There's no need to raise bug reports when automated tests fail because the CI server automatically notifies developers of failures instantly.
The true status of development is visible to everyone (not just developers) via the CI Server dashboard all of the time. Which means development can't offload poor code into testing to meet their deadlines when everyone can see the tests are still failing.
Any regression tests that need to be updated due to changes in the software will be discovered during the development phase, giving testers plenty of time to amend them before the testing phase starts.
You now have a completely integrated automated build and testing framework that can be extended to automate other tasks, such as the deploying releases and patches to test environments.
This does require that your development teams use a source control management system and have an automated build process in place. If they are unwilling or unable to put these in place or to integrate their build process with your continuous integration server hire a build engineer for a few days/weeks to set it up for you. You don't even need access to the source code as long as you can obtain the latest packages of your software.
Step 5: Reap the rewards
Put this all together and now not only does your test plan look like this, so does reality:
You have more time to test new functionality, spend less time reporting failures and because every change is fully regression tested during testing and development the days of finding critical bugs late in the testing phase and blowing out the schedule are over. You're testing more, in the same amount of time, but with greater visibility and earlier feedback. You might not be holding daily stand-ups, using task cards and working in iterations - your development methodology might not be Agile, but now your testing definitely is.
Andrew Prentice discusses adapting to an agile environment
A couple of months after I blogged about the role of the Disturbed in the Confluence team, it has come around to being my turn to take on the job. At stand-up this morning, we realised there's a pretty strong parallel between what the Disturbed does, and playing World of Warcraft.
A tank is a character whose primary role is to absorb damage and prevent others from being attacked. Tanks are "meatshields", so to speak, putting themselves between the mobs and the more vulnerable party members. The tank assumes the aggro of the mobs and tries to keep them off other group mates. With everyone nuking the same target (directed by the main assist), the mobs go down faster and there is less damage done to the group. (Source: WoWWiki)
So that's what I'm doing for two weeks. Holding aggro.
Gathering customer input is a critical part of a Product Manager's job. Atlassian PMs have an ace up their sleeve: http://jira.atlassian.com. We've opened our internal bug tracking system to the world, and use that to help understand our customer's needs.
You Want Me To Talk To How Many People?
Talking to every customer is easy when you're a brand new startup with a handful of customers. It gets harder as your customer base grows, and if you're a volume business like us - we have 15,000+ customers - it's completely impossible to have a separate conversation with every customer. Your customers are going to talk about you - on forums, blogs, mailing lists, and Twitter. If you have a popular product, trying to keep track of all these conversations and distill them into engineering work would leave you with little time for anything else. For example: sleep.
One way of dealing with this is to open your internal issue tracker to the world. Unsurprisingly, we use JIRA , but you can use whatever suits you. Our customers can create issues "I want a whizzbang", comment on them "It'd be cooler if it worked like that", and vote on issues that are important to them. They can also watch an issue, which means they'll get an email anytime an issue is updated - when someone else comments, when we start work, and so on. If you've ever worked with or on an Open Source project, this might sound familiar .
From a product manager's perspective, I get some wonderful benefits: I can see the issues that matter most to my customers , I can view detailed descriptions of the problem they're facing, and I can comment or email directly when I need more detail. I no longer need rely on sales people to help me understand a customer's perspective.
With Great Power Comes Great Responsibility
As with all powerful tools, this data can be dangerous if not treated with caution. The people posting and creating issues are your most passionate users. This is a wonderful situation to be in, but you need to be aware that this is only a subset of your customer base. You're not going to get much feedback from prospects and from people that aren't in your current market.
Secondly, since you're providing a forum for your customers to talk to you, you'd better be listening. Don't create expectations you can't meet - pay attention and explain why you're doing A instead of B. If you don't, this happens . As you can see, we've been far from perfect on this ourselves... but we're working hard to get better.
Finally, if your developers don't use the same issue tracker, it's pointless. You'll end up with one issue tracker full of customer requests that's genially ignored, and a pissed-off customer base.
I Wouldn't Work Without It
I'm based in Sydney, Australia. Our customers are in 130 countries around the world. There's absolutely no way I could know as much as I do about our customers' needs, wants, and desires as I do without an open issue tracker. Give it a go. It'll change the way your company works, and your customers will love you for it.
Shipping a new feature-version of software every few months - instead of every few years - is great.
Short release cycles help avoid feature creep because there is always a deadline looming, and these deadlines help to maintain a sense of urgency in developers and in stakeholders
Short release cycles help you avoid the mad rush of things that "absolutely must be in this release": The next release will never be too far away, so you'll have better chances of convincing stakeholders that non-critical features can be delayed;
Short release cycles force you to fix bugs regularly - every developer would obviously love to spend all their time working on cool new features instead of fixing bugs, but if you ship frequently you can't procrastinate forever. Stuff just has to work;
Since the release day is the best day to step back and see the big picture (and the competition), short release cycles help you to adapt your strategy to reality much faster;
Last not least, seeing progress frequently is great for team morale.
But you can make many mistakes in those few months too.
Unfortunately, even within a short release cycle of just three months, there is plenty of time to introduce new problems too. A typical team could probably design three totally unusable new features and introduce five unexpected performance bottlenecks along the way, on top of normal bugs. So three months can be a pretty long time as well, and usually you will only realise these problems around the launch date! Do you really want to wait this long?
Admittedly, there are some agile risk mitigation strategies: the continuous integration server will catch quite a few bugs and help the developers fix them, automated performance tests let you discover many subtle problems really early and the QA team will be working endless hours to iron out usability issues too. But let's face it: even the best QA and usability experts are just simulating user behaviour. The real test is when actual users work with your software, in ways you maybe didn't consider, and when your software interacts with other production software too. The earlier you can get this data the better.
How we do it at Atlassian
Enter Dogfooding
Dogfooding means nothing else than using your own products. This is a great principle because by using your own software, you will share similar pain to your customers. If your product is crap, you will have a high incentive to make it better, if you use it every day. You will make features work well, not just look well. You will fix bugs before you add even more features. No matter what software you are developing, make sure to use it yourself in one way or another.
According to Wikipedia, the term was coined by an actual dog food advertisement, and later promoted by Microsoft. Even if this remains a myth, the term is very catchy. Eat what you sell! Use the software you ship! Stick your own finger into a table-saw to demo the shutdown mechanism! Well, you don't have to go this extreme to become a good dogfood-er. Just deploy your software onto a system that you have reason to use yourself. You may need to become creative and bend some company rules, but few things can't be done when the stakes are high. Are you developing yet another Twitter-clone? Then shut down all Twittter access for all employees from within the company. Are you coding a new Wiki? Throw out Confluence! Are you testing dull online-banking software? Try moving actual money onto a staging-installation to encourage employees to spend that money under realistic settings. There will be pushback in any of these examples, but it is usually worth the effort to fight this through.
Mini-releases to dogfooding servers
So, rolling out your final product to yourself is great. But just shipping your final release to your own company doesn't make your product better in the short term. If the final release is crap, then it won't be just your customers who are mad at you, but your coworkers and boss too! On the one hand you do want their feedback from dogfooding, but on the other hand you don't want to upset them by putting a dysfunctional final release in front of them. The solution is pretty simple: ship to your own company servers all the time. Not just once per quarter, but every two or three weeks!
At first, that idea might sound even worse, because now you might have your boss get angry at you every two weeks instead of every three months. But the good news is that you'll get a chance to improve the product right away, once he or she has stopped yelling at you. Bug fixing and improving unusable features will bubble up in priority almost immediately after a botched dogfooding release. Additional features will only be worked on once the existing features are good enough. Automatically, the next dogfooding deployment will cause much less frustration. And the next will be even better. Until you fall into a rhythm of always shipping decent dogfood releases. It just happens!
Feedback from staff about mini-releases
On top of improving quality, frequent internal releases are great for gathering regular feedback as well. Inside Atlassian we upgrade our Confluence servers every two weeks, and we even create release notes for each release, including screenshots and explanations. Creating these release notes is simple and just takes an hour or two, but the feedback from our internal staff is amazing. Seeing the release notes, and the actual mini-release, every staff member knows what changed and what to comment on (if they care). The development team and the product managers get free feedback from actual users, all the time. So it is a very worthwhile investment for the development team to build and deploy regularly.
Here is an excerpt of one recent mini-release (we call them milestones). You can view all our milestone release notes online too, because we even publish them to our development community:
And here is an excerpt of the feedback we got from various parts of the company. Bill works in Marketing, Joshua in Sales, David in IT, and Matt in Development. These are just 6 comments out of 26 we got, and that is just one single milestone release being discussed. We get this kind of feedback all the time, and get even more by chatting in the hallway or through IM or JIRA issues!
While QA is really important in helping find bugs and unusable features early, only real users (and as in this case: customer representatives) give you the whole picture. If we didn't ship internal Confluence releases all the time, we would get this kind of information far too late to feed it back into the upcoming release.
Trying this at home
We are extremely happy with our dogfooding process in the Confluence team, and we are step by step rolling it out to our other product teams. However, before you get too excited, have a look at some risks and common sense prerequisites
Risks
As mentioned before, dogfooding does carry certain risks. Despite almost two years of fortnightly deployments, we still sometimes run into some common problems:
we still sometimes over commit during a two-week iteration. We sometimes ship features to our internal systems that should have remained on a developer's machine for another two weeks, because it is just not useful yet;
while we do have a great QA team, sometimes we just don't listen closely enough and ship more bugs to our internal audience than we should;
sometimes we find so many bugs after shipping internally, that we can't get all the bugs ironed out before the next milestone, and some of our internal users occasionally get annoyed.
But then again, we do find problems while dogfooding that we would not find even with three times as many testers. Occasionally marketing or sales employees do get a bit mad at Confluence developers for introducing a new bug on our intranet server. But everyone agrees that bugs they stumble over at least won't make it to customers. That's great to know for marketing and sales employees too.
And a completely non-technical risk: Even though we try, we don't use all the features internally in the same way our customers do. We must always ensure we don't get lulled into a wrong sense of security. It is crucial the software works for us, but it isn't sufficient. Having paranoid evil testers remains as important as ever.
Criteria for dogfooding deployments
Since you will run into some problems, it makes sense to define a couple of simple criteria for deploying to your internal dogfooding system. You have to strike a balance between stability and dogfooding value. If you focus too much on experimentation you will get lots early feedback, but it will be mainly negative feedback. If you focus too much on stability, you will avoid negative feedback about bugs, but you will get the feedback so late in the release cycle that you may not be able to improve the feature based on the feedback anymore. In Confluence we tend to have these criteria:
all automatic tests (there are thousands!) on our Bamboo servers must pass for the target configuration (if we deploy to Tomcat, then the Tomcat-tests must pass);
the automated performance test must not show a significant slowdown;
deployment to an exact replica staging system has to work, and manual smoke testing of the top use-cases must not show any problems on that staging system;
the new features must have been checked by QA, no showstoppers have been found, and the amount of known bugs must be acceptable. We don't aim for zero bugs in mini-releases though! We want to know early if we are on the right track with a feature. A few bugs more or less don't matter as long as end users can see the general idea of a new feature. Insisting on zero bugs would take a long time, and then the crucial feedback might come way too late.
Public dogfooding
We also run our unreleased versions of Confluence on our publicly visible servers. However, we don't run our fortnightly releases there. Since the focus on internal dogfooding is to find bugs and usability problems early, we don't have extremely high requirements for polish for the milestone releases. But our external systems are also used by potential customers, and we don't want them to think that interim bugs will actually be in the final software release. So our external dogfooding is mainly done with patch-releases (which don't add any new functionality) and with beta-quality releases during the last month of a release cycle. We still take a certain risk that bugs can occur, but it is much more controlled. Getting feedback during the last month of a four-month release cycle is pretty late, but this still gives us enough time to react to some user feedback and to real-life problems you wouldn't encounter on internal systems (e.g. SPAM, search engine web crawlers, anonymous access, problems with hackers, etc).
Be pragmatic and customise to your needs
Too often agile projects have gone wrong because people thought they could blindly follow some approach they read about on the net. Don't be naive! It has taken us a lot of thinking and careful preparation to get to where we are. We wouldn't be as successful, agile, and fast-paced without our fortnightly dogfooding feedback. But this doesn't mean one size fits all, and that everyone should throw all their process over board over night and follow our example. Take careful steps and precaution before you try this at home! Backups help. QA helps. Trying it out on non-production systems helps. Discussing it with your team is crucial for success. Clear communication is important as ever. Don't try this in the middle of a failing project either, rather wait and introduce this process after a major release has taken place. And so on. With some common sense though, continuous dogfooding will make you wonder how you ever managed to develop software without it!
Watch Per talk about his approach to agile iterations
Copyright © 2009 Atlassian Pty Ltd.
