Lemme start by saying two things to regular readers: 1) “Thanks for tuning in and dropping lots of thoughtful comments!” and 2) “You’ve probably noticed me getting all up in your area codes about fast feedback lately, what with the artifact sharing and inner/outer loops n’ all.”
Yeah, it’s been a bit of an obsession ever since the Agile2012 conference got me thinking about how important it is for agile teams in particular –partly because turbo-charged feedback is a natural work-in-progress limiter. How? Getting results back to devs so quickly that they don’t have time to start in on another issue keeps everyone focused on the task at hand. That, in turn, fosters an efficient development process. And efficiency, in my never-humble opinion, is what agile development is all about.
But before I put this feedback thing to bed for a while, I want to dig back into my long-lost career as a test automation engineer and share with you the technique we used for running tests in parallel. ‘Cuz it was super-easy and I loved it.
TestNG Groups FTW!
If you’ve never used TestNG before, you might think that’s a typo above and that I meant “TestING”. (Google’s autocomplete certainly thinks it’s a typo, so hey: you’re in good company.) But no. TestNG is an automated testing utility, much like JUnit. In fact, you can use it in combination with JUnit, WebDriver, Selenium, etc. And the reason you’d want to is Groups. By sorting your test suite into groups, then creating separate jobs in Bamboo to run each group, you can speed up your testing stages by 100% –and likely, better.
TestNG is designed for Java tests, but can also be used for Ruby projects. And it’s not the only framework to offer groups. Proboscis for Python and PHPUnit also have them. JUnit has “categories”, which largely serve the same purpose, but are rather clunky and make TestNG’s groups look like a bacon-wrapped Ferrari by comparison (sleek, fast, and yummy).
Implementing Groups in Your Test Code
Like a handful of other testing frameworks, TestNG takes it’s cues from an @Test annotation. Inside that, you can place a number of attributes, one of which is Groups. Group names are both dynamic and arbitrary –small sentence, big implications. First, you can create a new group on the fly just by assigning a test to it. No need to declare and define the group in a separate file. The act of assigning a test to a group will also create that group, if it doesn’t already exist. (Now, that’s a double-edged sword because it also means you can end up with a group called “smoke_test” and another called “smoke_tets” if you’re careless.)
Second, you can devise whatever system (or systems) of meaning you like for your groups. Slice them along functional area of the application, by level of the code stack, by whether you currently expect the test to pass, etc. And tests can belong to multiple groups. So a single test might conceivably belong to the “user_authentication”, “API”, “smoke_test” and “in_development” groups. TestNG even supports groups of groups. I haven’t played with that personally, but it’s pretty bad-ass and their documentation on it is helpful.
Quick protip: TestNG doesn’t like it when you use a hyphen in group names. Use the underscore instead.
Of course, if your application’s super-structure doesn’t know about TestNG, you’ll get all sorts of errors and your builder won’t know what do with all those groups at runtime. So be sure to declare TestNG as a dependency for your project or module.
Using Groups in Bamboo Builds
Back in Bamboo, set up Jobs based on the groups you’ve defined in your test code. If you’re executing tests via Maven, Gradle or Ant, this is pretty trivial. In the example shown here, I pass a group name into Maven as an argument. When it’s time to run the tests, TestNG will scour my project, pull out all the tests belonging to that group, and run only those. Doesn’t matter if the tests for that group are located in different classes or packages. As long as they’re in the same project or module, TestNG will find them.
Note that because each Job has it’s own working build directory, you’ll need to pull the test code into each Job. A source code checkout Task is generally the simplest way to do that. If your code base is large, you can take advantage of the option to check out from a sub-directory in your repo and only pull down that part of the tree. Another option is pulling down all the test code in the very first Stage of your Plan, then zipping it up and passing that as an artifact to downstream Jobs.
Agents of Change
It’s probably obvious, but splitting your test suite into 10 batches isn’t going to help you much if you only have 1 or 2 build agents. Of course we’d love if if everyone added more agents to their Bamboo license so they can really take advantage of parallelization. Maybe that’s feasible for you, and maybe not. Either way, be mindful of your agent count when considering how to split up your test suite so you’re not just spinning your wheels.
You can take this idea of parallelized tests about as far as you want. There’s no restriction as far as TestNG or Bamboo is concerned. Many of our own CI builds have over 20 test groups running simultaneously. Again: you’re limited only by the number of build agents you’re running.
A Parallel Universe
As with most things build eng, there are several ways to accomplish parallelized testing. Tinkering with custom scripts or your suite.xml are two other common approaches. For big multi-module projects, it might make sense to split the suites/groups into separate repositories and run a Job for each repo. Regardless of which method you chose, you’ll get the most bang for your buck by focusing your parallelization efforts on integration-, API- and UI-level tests. Unit-level tests already run quite fast, so they’re unlikely to be the bottleneck in your builds. (Extra credit for the over-achievers: many testing frameworks allow you to run unit tests multi-threaded!)
Much of the fast feedback stuff I’ve been harping on lately comes around to build agents in one way or another. If it’s been a while since you thought much about them, check out the new agents section in Bamboo’s Quickstart Guide for a refresher.