Recent posts from John Ferguson Smart

This Bamboo customer story is the final of an 8-part blog series about why so many developers adopt continuous integration written by our guest blogger, John Ferguson Smart.
John is a consultant and trainer specialising in Build Automation, Enterprise Java, Web Development, and Open Source technologies, currently based in Wellington, New Zealand.

Continuous Integration for Ruby

Jeremy works in a large company specialized in Linux security and backup applications. This company is a Ruby shop - their applications are written in Ruby on Rails, with the source code is stored in Subversion. These applications include Linux software packages that automatically back up the system to a tape drive. This sort of low-level software is hard to test, as the software package needs to be deployed and installed onto a remote machine before comprehensive testing can be done. However, as with any other software product, automated testing is nevertheless essential for the overall quality of the software.

Jeremy uses Bamboo to automate the build and testing process. His Continuous Integration environment involves both running Ruby builds and tests on Bamboo, and also automated deployment and installation on a remote test machine. To maximize responsiveness and minimize network load, Bamboo builds are triggered by a Subversion hook script set up in the source code repository.

Easy configuration

Bamboo integrates well with languages other than Java, including Ruby on Rails. With a little Bamboo customization, Ruby on Rails can be treated as first-class citizens in the world of Bamboo build jobs. Bamboo does not recognize Ruby out of the box, however. The first thing Jeremy had to do was to add a new "Command" build capability, to let Bamboo know where to find the ruby executable:

Once Bamboo is configured, it is easy to create new Ruby on Rails build plans. Ruby-based build plans differ very little from ordinary ones. The main obvious difference is the use of the new Ruby capability in the Builder tab when you create the build plan:

Testing practices are valued in Ruby circles just as much as amongst Agile proponents in the Java world, and there are is good support for testing in Ruby. Like many other Ruby projects, Jeremy uses Test::Unit and RSpec for unit testing and Behaviour-Driven-Development style functional testing. Both these tools produce HTML reports which the Bamboo build plan stores as artifacts, allowing Jeremy to browse them afterwards.

Integrated reports for Ruby

Jeremy also uses the ci_reporter tool to convert the test results from these tools into a JUnit-compatible XML format, which can be easily understood by Bamboo, providing integrated reporting on test results and durations. The rails_rcov Rails plugins provides code coverage data, and can be integrated into Bamboo via the RCov plugin for Bamboo.

Automated tests are executed from Bamboo in three phases. First of all, unit tests are run on the Bamboo build server, directly on the code checked out of the version control system. The second phase, involving functional testing, is more complicated. The built application needs to be deployed to a test machine, and installed and tested there. This is done using SSH - the software is deployed and the test machine rebooted. Bamboo waits for the test machine to reboot, and then runs a series of functional tests on this machine remotely from the build server. Finally, a series of tests is run directly on the test machine.

What's your adoption story? Tweet Atlassian or leave a comment below about how and why you adopted CI.

This Bamboo customer story is the seventh of an 8-part blog series about why so many developers adopt continuous integration written by our guest blogger, John Ferguson Smart.
John is a consultant and trainer specialising in Build Automation, Enterprise Java, Web Development, and Open Source technologies, currently based in Wellington, New Zealand.

Clever tools to optimize Continuous Integration

Sandra works in a large government service in Toronto. She is in charge of a team of seasoned developers whose job is to provide technical support and mentoring to other teams, mostly made up of less experienced developers. Recently, her team has been called in to help out a team full of very inexperienced Java developers on an important external-facing web application. These developers, recently brought over from COBOL and the world of mainframes, have a basic Java training but very little experience in the way of best practices. Her team's job is to mentor these inexperienced developers, teaching them the tricks of the trade, so to speak.

Appealing plugins

Metrics play a key role in this task. Sandra set up a special "code quality" build plan, which runs a swathe of static analysis tests (including Checkstyle, PMD, and FindBugs), and also comprehensive code coverage reporting using Clover. The Bamboo plugins for each of these tools give her a high-level picture of how these metrics evolve over time. This is an important tool for her to keep tabs on how well the trainee developers are applying the techniques that they are being taught. For example, low or dropping code coverage on a class may indicate that one of the new developers is having trouble assimilating the test-driven development and testing practices that her team is trying to teach.

Although a high-level view is just what the doctor ordered when you want to keep tabs on the overall health of your development practices, there are still times when you would like to be able to drill down into the details, and see exactly what the violations were in a particular build, or what lines of code were not being tested. To this end, Sandra stores the HTML reports generated by each tool as build artifacts. This way, she can drill down into the detailed reports of any build whenever she wants.

Watertight reports

These reports are also an important part of Sandra's training strategy. She reviews both the high-level statistical views and the detailed reports in a weekly code quality meeting, where any new issues or trends are discussed. In one of the first meetings, for example, Sandra pointed out the increasing number of Checkstyle violations related to an absence of Javadocs. A short talk on the merits of commenting ones code was enough to put most of the developers back on the right track. The issues raised by FindBugs, for example, are particularly educational, and are often used to discuss some of the finer points of Java programming. In one instance, for example, the discussion over one FindBugs issue reveiled a fundamental misunderstanding about the immutability of Java Strings, which was rectified on the spot by an impropmtu tutorial on the topic of Java Strings.

What's your adoption story? Tweet Atlassian or leave a comment below about how and why you adopted CI.

This Bamboo customer story is the sixth of an 8-part blog series about why so many developers adopt continuous integration written by our guest blogger, John Ferguson Smart.
John is a consultant and trainer specialising in Build Automation, Enterprise Java, Web Development, and Open Source technologies, currently based in Wellington, New Zealand.

Automating the staging and production deployments

Testing intermediate builds is one thing, moving to General Availability and production deployments are a whole other subject. Steve and Simon, the developers in story 5, also had this issue to grapple with as prior to using Bamboo this was an entirely manual process. When they moved to using Bamboo for Continuous Integration they also upgraded their issue tracker to JIRA which made for a very pleasant surprise.

The pain of manual builds

Originally, this was a manual process, requiring a developer to run the mvn release:prepare and mvn release:perform commands from the command line. Maven would prompt for the next version number, which had to be entered by hand. If an error or mistake occured, or the process was interrupted half-way through, things had to be tidied up by hand. Many times, an incorrect version number was deployed, and the Subversion tags had to be removed by hand afterwards. In addition, an obscure Subversion bug meant that occasionally the release:prepare phase would fail for no reason. Although the work-around was relatively simple (just run an svn update and re-run the release:prepare command), this increased the complexity of the process and added one more thing that could go wrong. It also duplicated maintenance work between the version numbers in JIRA (where issues were managed) and in the Maven project itself.

Preparing Maven, JIRA and Bamboo

In Steve and Simon's project, Steve now uses JIRA and Bamboo to help automate the build promotion process. The version numbers of each release are maintained in JIRA. There is generally a maintenance release once a week, on Friday night, and a major new release every two months or so. Version numbers use the default three-digit Maven convention (e.g. 2.0.44). Each version released to staging is given a unique version number. Automated deployments to the integration server use snapshot versions (e.g. 2.0.45-SNAPSHOT).
Since their project uses Maven, he uses the Maven Release Plugin to help manage the build promotions. This powerful plugin checks that all the latest changes had been committed to Subversion, creates a new tag in the Subversion repository with a label matching the next release number, and updates the Maven project version number. In a second phase, the tagged code is checked out, built, tested and deployed to the Nexus repository, ready to be used during the staging deployment.

Using Bamboo and the JiraVersion plugin, Steve was able to entirely automate this phase. The Maven Release plugin can be run in batch mode, without any human interaction. By default, it will simply increment the current version number. However, you can also provide the next build number as a parameter. This makes it easy to integrate the JiraVersion plugin, which will pass the selected version number as a parameter to your build script. In Bamboo, Steve set up a build plan configured with the JiraVersions plugin. This plan proposes a list of available JIRA versions for deployment. Steve simply chooses the version that he wants to deploy to kick off the build promotion.

Practice makes perfect

With a little work, Steve was able to automate the process and create a series of Bamboo build plans that authorized users can now use to deploy to the right environment with the click of a button. In Bamboo, there are now two special "release" build plans, one for staging and one for production. The staging and production build plans are run on remote agents, directly on the staging and production servers. Each build plan can only be executed by people authorized to deploy to staging and production respectively. These build plans propose a list of available JIRA versions for deployment, and the user simply needs to choose the right version number to kick off a full-blown deployment.

To get around the Subversion bug mentioned above, Steve wrote a Groovy script that automates the whole process, including the work-around if the Subversion issue occurs.

Now, whenever a release is ready to go onto staging or production, the build promotion process is as simple as clicking on the corresponding link in the list of available version numbers.

What's your adoption story? Tweet Atlassian or leave a comment below about how and why you adopted CI.

This Bamboo customer story is the fifth of an 8-part blog series about why so many developers adopt continuous integration written by our guest blogger, John Ferguson Smart.
John is a consultant and trainer specialising in Build Automation, Enterprise Java, Web Development, and Open Source technologies, currently based in Wellington, New Zealand.

Optimizing Continuous Integration to push build artifacts

Simon Smith and Steve Peters are Java developers working on a large web application alongside 5 other developers, spread over two geographical sites. The application is designed using a clean modular architecture. The project uses Maven 2, which helps define and enforce the modular architecture. It consists of ten distinct Maven modules representing different parts of the system, with well-defined dependencies between the module. For example, there is a Maven module for the core domain logic, and another for the web layer. Steve has also written a specialized library encapsulating some customized localization features. Source code is stored in a Subversion repository. Build artifacts (JAR and WAR files) are deployed to a Nexus Maven repository, to be made available to other developers.

The Maven repository distinguishes between snapshot versions, which typically correspond to a nightly build or to the latest code changes made to the version control system, and release versions, which are more stable, numbered releases. Developers in the same team will typically use snapshots reflecting the latest code changes, whereas release versions are usually for external use where more stability is required. This approach does an excellent job of enforcing modularity and keeping dependencies clean.

Life before Continuous Integration

However, before CI was introduced, their approach to build artifacts was a source of frequent integration headaches. For example, if Steve corrected a bug in the localization module, he needed to first commit his code changes to Subversion, and then to manually deploy a new snapshot onto the Maven repository. Only then would his corrections be available to Simon. The deployment phase was time-consuming, as it involved executing a complete build and test cycle, and then transferring the file to the Nexus server on the other geographical site - depending on the size of the module, it could take between 5 and 20 minutes, several times a day, for the whole process to complete. Steve would occasionally forget this second step, which resulted in his corrections not being visible to Simon, even though they had been committed to Subversion. This also happened occasionally in the nine other modules.

The main impact of this sort of incident was lost time. The problem was relatively insidious, as in theory at least there was a simple work-around - Simon just needed to check out Steve's module and do the deployment himself. However this would interrupt Simon in his thought-stream, and oblige him to work on something else for 15 to 30 minutes. The problem was also hard to isolate: one time, it took 4 hours of debugging before Simon finally tracked a problem down to an undeployed snapshot. And if something went wrong, Simon was hard-put to trouble-shoot the issue.

Switching to Bamboo

This sort of wasted time prompted the team to look for a more automated solution. So, to make the overall integration process smoother, the team decided to install a Bamboo server. There is now a dedicated set of build plans for each Maven module, that compile and test the code whenever changes are made, run code quality and code coverage metrics, and notifies developers if something went wrong. In other words, from a CI perspective, each module can be considered as a separate CI project.

Since the development efforts are now closely integrated, modules use snapshot dependencies to ensure that they always have the latest code changes. Each module also has a second build plan, which automatically deploys the build artifact to the enterprise repository whenever the first build succeeds. Since the initial build runs a battery of unit and integration tests, a minimum of stability is ensured. For Steve and Simon, the result has been a massive time saver. Now, every developer can now access the latest stable versions of the libraries with no manual intervention required. The deployment still takes 5 to 20 minutes, several times a day, but it happens automatically, with no human intervention, and with no interruption in the developer's concentration.

What's your adoption story? Tweet Atlassian or leave a comment below about how and why you adopted CI.

This Bamboo customer story is the fourth of an 8-part blog series about why so many developers adopt continuous integration written by our guest blogger, John Ferguson Smart.
John is a consultant and trainer specialising in Build Automation, Enterprise Java, Web Development, and Open Source technologies, currently based in Wellington, New Zealand.

Driving quality with hyper complex distributed builds

Sophie is a technical project manager in a large insurance firm. She manages the development of a web-based calculator for car insurance premiums. The calculations are done dynamically, using a sophisticated AJAX-based web interface. Car insurance premiums use complex algorithms, and it is vital that the web calculator results match those coming from the mainframe back-end to the cent. As a result, the application has a very large battery of unit tests - some 6000 unit tests are run against the calculator alone. The application also has web-based functional tests, using Selenium, and load tests using JMeter. Each build of the application needs to be tested against four different browsers (IE6, IE7, Firefox, and Safari). The team has also written a comprehensive test of integration tests which run against a production-scale Oracle test database and a CICS mainframe back-end.

All together, the full build can take up to an hour to run. Far too long for an effective Continuous Integration set up.

There is also another problem: although there are effectively Selenium test scripts that run smoke tests and functional tests against the deployed application, they need to be run in several different browsers, and on different operating systems. All of the targeted browsers can run on a Windows platform, but the department build server is set up on Linux box, and the QA department want the automated Safari tests to be run on an OS X machine.

The case for Continuous Integration

Distributed builds provided a good answer to both of Sophie's major problems. A distributed build architecture lets you run build jobs across several machines, providing potentially huge performance gains. It also enables you to run specific builds on dedicated machines, which makes it possible to run OS or environment-specific tests or build steps.
To resolve these problems, Sophie used a distributed Continuous Integration environment based on Bamboo. The CI setup consists of a main build plan, which compiles and runs the unit tests, along with build plans for functional tests running against different browsers (IE6, IE7, Firefox, and Safari), as well as other plans for the load and integration tests. Finally, there is a separate build plan for code coverage and code quality metrics. Seven build plans in all.

In Bamboo, it is a straightforward task to set up remote agents for your distributed builds. The main Bamboo server coordinates the builds, and runs build jobs either locally or on remote agents, depending on their availability. You can also use "capabilities" to help decide where a particular build job should be run. For example, Sophie has set up build agents on 5 machines: 2 recent and powerful Linux machines, 2 older Windows XP boxes, and a brand new iMac. The Firefox tests can be run anywhere, but the IE tests need to be run on the Windows machines, and the Safari tests are to be run on the Mac. She has added a custom capability called "operating.system" to help Bamboo know where to run each of the functional test build plans. Then, all she needs to do is to add an extra Requirement to each build plan to indicate what operating system that particular plan needs.

Load tests are done by deploying the application to a dedicated test server and then running a JMeter script on the build agent. Load tests are particularly processor-intensive, and for best results need to be run on fast machines. Sophie only wants load tests running on either one of the fast Linux machines, or the iMac. To do this, she has added another custom capability called "high.performance" to identify the machines with enough power to run the load tests correctly.

Syncing the build artifacts

One other issue that Sophie had to resolve was distributing the build artifacts. The normal operating procedure in most CI setups is to check out the latest copy of the source code, compile and build the application, and then run unit tests, integration tests, and so forth. In Sophie's case, this would mean that the application would be compiled, unit tested, and bundled up into a WAR file seven types for each build!

Sophie's project uses Maven 2, so she was able to use Maven's support for snapshot releases and dependency management to optimize things in this area. The initial build plan compiles, runs unit tests, generates a WAR file, and deploys the WAR file to the Enterprise Maven snapshot repository, making it available for the other build jobs. This initial build is the only build to be triggered off by changes in the source code repository - all of the other builds are dependent on this one. The integration, functional and performance tests are set up as separate Maven projects, thus avoiding the need to rebuild and rerun the unit tests each time. These build jobs automatically download the latest application snapshot and run the appropriate tests against this version. For the functional tests, Maven profiles are used to determine which browser to use in each case.

What's your adoption story? Tweet Atlassian or leave a comment below about how and why you adopted CI.