We Are From Mars

All agile methodologies stress the need of co-locating development with the customer's representative - the Product Owner - or at least, having them in close proximity - both in time and in space.

Our team seems to be the exact opposite of what "the agile way" requires - we are located on the other side of the planet (Gdansk, Poland) from the company headquarters (Sydney, Australia). This means we could just as well be located on Mars. Actually, one could argue that outside of the narrow communication windows (early morning and late at night), communication latency between Sydney and Mars is smaller than the one between us and our Product Owner. Yet - we still manage to work in an agile way.

Talking To Each Other

How do we do it? First of all, we utilise high-bandwidth communication opportunities as much as possible. We have a local Product Owner proxy (that would be me), who talks to the Product Owner regularly, at predefined times and at least once a week. Also, if the out-of-band need arises, ad-hoc meetings are scheduled. Skype (or some equivalent technology) is your friend here, as well as Instant Messaging systems, such as Jabber. I cannot stress this enough - talking to each other in real-time is the most important and most effective way to communicate. I do not think it would be possible for us to work without it.

Technology Helps

But that is not all. For us, the second most important communication method (and tool) is JIRA - and it does the job just fine in its plain, out-of-the-box form. We use pretty much a stock JIRA setup, with just two custom fields added (both reflecting standard Scrum practices) - one called "backlog order", used for prioritizing user stories (a fancy word for "new feature or improvement"), the other called "story points" - for estimating the "size" of the user story. That suffices all our needs for making sure both the PO and us know what has to be done and how much delivering the requested feature is going to cost (after all, the effort estimates translates almost linearly into monetary costs). We also use the GreenHopper JIRA plugin to provide "virtual story cards", instead of maintaining our user stories on pieces of paper, to make our life easier when it comes to copying the outcomes of our planning sessions to JIRA. But for day-to-day ticket management, we just use JIRA. Or more properly: one of us (that would be me again) is using JIRA, everybody else is just moving around the actual paper story cards on the cork board, which is a central part of our workspace.

What about the communication in the other direction? What if a Product Owner needs to describe some story to us, and words are not enough? Sometimes the Product Owner will want to draw a mockup of the user interface he has in mind. Here is where the Balsamiq Mockups tool is very helpful. It is a Confluence plugin, which lets you draw diagrams right there in the Confluence page, and share it with whoever you want to communicate with. We would typically edit the diagrams in-place and simply comment on them until everybody is satisfied with the resulting sketch of the user interface to be delivered.

Delivery

How does the Product Owner know we are doing what he wanted us to do?

First of all, just as the Scrum methodology requires, we have a proper (internal) release every two weeks. The official builds are published in an externally accessible place for everybody to download and try it out - this gives us a way to regularly demo our progress and provides a way for the Product Owner to correct our course (e.g. whenever we stray from what has been planned or whenever the plan needs to be changed due to external circumstances).

Second - our Bamboo-based automated build system lets the Product Owner download and use the snapshot of our work every day - the snapshot may be barely usable, but typically it is in a good enough shape to let the Product Owner try new features as they are delivered.

Remote Dogfooding

We regularly practice dogfooding, using our product regularly on a daily basis. Almost every nightly snapshot is taken and used by each team member every morning. So if we happen to have a bug, we are quite likely to intercept it before the official version of the product is released. Also, if some user interface elements are awkward to use, or somebody has a bright idea for improving it, we are able to try multiple solutions and pick the best one - simply because we use our own software for our day-to-day work.

But in order to get even better feedback, developers in the Sydney office should also dogfood our software and give us feedback. Achieving sufficient levels of internal adoption requires a bit of advertising from the Product Owner, and regular presentations of cool new features that the product provides, so that folks in Sydney have a reason to bother installing our stuff.

Separation Of Responsibilities

The last thing that is important for a remote team and the Product Owner working with it is a separation of responsibilities. In our case, the Product Owner is only responsible for prioritizing features and controlling if they are delivered as promised. Planning and implementation are left to us. Also - all bug reports are handled locally by the team, as a part of the Quality Assurance process - we are solely responsible for keeping the product in good shape and the Product Owner does not interfere with that.

Another important rule we follow, is that the team's effort estimations are final. The Product Owner (or anybody else) does not even attempt to question them, trusting our judgement - in true agile spirit.

First, my definition of refactoring:

Refactoring is improving code without changing the features it implements.

That's all.

If you're refactoring, you're not fixing bugs, you're not improving performance and you not increasing robustness. Refactoring is simply improving the design of the code, while ensuring that it still works the same, warts and all.

Pointy haired bosses the world over froth at the mouth to hear such things. Veins pop out in their temples. You mean the business value of the software stays the same but the cost to the business goes up? I didn't say that.

If you measure business value only in terms of features you have today, then you can end up deep in technical debt; you can add features today in such a way that features tomorrow cost more and more. The value of refactoring is wholly contained in the future ability of programmers to comprehend and modify the code. It's called maintainability, but that's a boring word, so let's call it Agility. Mmmm, sexy. Well-factored code is agile code because it's better able to change.

In economic terms, refactoring is an investment, or the repayment of a debt. It's only worth doing over a time frame when the interest payments or repayments (in the form of ongoing productivity gains) compound to exceed the time invested. Fingers crossed the business or project sponsor is also planning over such time frames.

The term refactoring comes from mathematics. You may remember your high school algebra:

2x² + 10x

Stay with me! No glazing over! If you refactor the expression, extracting the common factor, 2x, you get:

2x . (x+5)

Sometimes it's hard to spot the common factors, in both mathematics and programming, and it can certainly be done poorly. More on that later.

Like most powerful techniques in software development, the purpose of refactoring is controlling complexity.

Complexity is bad, mmmkay. Complexity is evil.

I was fortunate enough to be chatting about project complexity with the legendary Dave Thomas (OTI, Eclipse) at JAOO Sydney in May. He nailed it: "kLOC kills". Complexity and scale in codebases is a major contributor to schedule blowouts, poor velocity and excessive development cost. Complexity is a kitten killer from way back.

Fred Brooks discusses two categories of complexity. Essential complexity and accidental complexity.

Essential complexity is the complexity of the domain. In NASA software, there's no escaping rocket science. You can isolate and divide essential complexity but you can never remove it. Essential complexity belongs to the problem. By contrast, accidental complexity is an artefact of the systems, languages, frameworks you're using. In principle it can be reduced by changing the system. Accidental complexity belongs to the solution. Refactoring reduces accidental complexity.

If you don't have much experience with it and you're looking for some concrete tutorials on refactoring, I suggest you start with Martin Fowler's seminal book Refactoring. Fowler also maintains a catalog of refactoring recipes with an Object Oriented flavour.

One of the most basic techniques is Extract Method which all decent IDEs can do automatically. You know you need Extract Method if you have a multi-page method with a sequence of of comment blocks which look like this: Now that we have the InductionActuator, look up the FluxCapacitor.... Doing it manually means snipping out a logical sequence of code and pasting it into a new, small, well-named method, stitching the local variables used from the originating context into parameters to the method. If this is hard due to sloppy scoping or too many variables, you may consider Introducing a Field from a local variable.

The inner loop of agile development should go like this: Red, Green, Refactor. Red means you have a test which is not passing. Getting the test to pass is the next step. Green means you are passing all tests. Refactor means... refactor.

Even if you're a good agile developer, doing things as simply as possible, complexity and duplication of common factors creeps in while you're trying to pass tests. Everybody hacks. Everyone copies and pastes. This is fine as long as you go back and refactor when you've got the green bar. Sometimes you may need to avoid mentioning this to PHBs, for their own good. Shhh!

Unit tests are really important for refactoring. If you're not doing unit testing you've got a long way to go. A good unit test suite is a necessary precondition for confident, aggressive refactoring. And IMHO a good test system is a necessary precondition for confident, aggressive, automated refactoring. These preconditions can present a quandary for some developers. Legacy systems often have no effective automated tests. And since they're often composed entirely of spaghetti, they need to be refactored. It's a chicken and egg situation, where do you start? All I can say here is you start small.

Refactoritis

Can you have too much refactoring? Absolutely. If you're somewhere around middle-stage zealotry for this refactoring stuff, you may not be in danger of copy+pasting your way to a big ball of mud, but you may fall prone to exceed the safe working abstraction load of your language or go too far beyond the idioms of your team's codebase or comfort.

Every language has limits imposed by its design and implementation. In Java and C#, for example, the limits are seen by many in the dynamic languages camps to be too much to bear. For example, say you're refactoring some Java or C# code. You might create a new interface with a few alternate concrete implementations and, whereas before you had two methods on a concrete class and a few big if-else blocks, after refactoring you might have three files and more actual lines of source code. It can be somewhat subjective but sometimes you may have more complexity even though you've removed duplication!

If this happens you have fallen asleep on the refactoring train and missed your station. Often you should just roll back the code and go write a feature. Some duplication is easy to see and cope with, especially if it can fit on one screen and any reader can see the pattern. In other languages, Lisp comes to mind, there are constructs (like macros) which allow you to encapsulate expressions that cannot be elegantly factored in, say, Java. Disclaimer: IANALN; I Am Not A Lisp Nerd.

So the expressiveness of the language can constrain refactorability. Another way of saying this is that the language contains accidental complexity and only factoring out the language can remove that complexity. I should say here that I have recently found Groovy to be a great candidate for doing this on Java projects.

As a more concrete example, lexical closures are a great way of implementing things like the new for loop (for each) introduced in Java 1.5 and functor frameworks that employ anonymous inner classes in Java for similar purposes (e.g. composable transformers, ad hoc iterator delegators instead of explicit looping) often feel too cumbersome compared to most closure implementations. So you just have to suffer the duplication and code bulk.

So in summary, Red, Green, Refactor, don't go overboard and be aware when your language makes capturing factors you see in your system worse. Kill complexity before it kills you.

If you're interested I'll be telling war stories and going into some side issues over on my personal blog.

At Atlassian we're always looking for ways to expand the utility and functionality of our products. Sometimes this means we develop that code ourselves, and sometimes it means our community fills in the blanks. Such is the case with a new integration we've been working on with Alfresco, an open-source content management company, and Sourcesense, a mutual partner of Atlassian's and Alfresco's. In a nutshell, we're integrating Alfresco with Confluence to bring OpenSearch to Confluence wiki pages and to embed Alfresco-stored documents into Confluence, among other things.

One of the best things about this integration is that it speaks to both the power of Alfresco's open-source code and Atlassian's open APIs: the integration was started without any formal communication between the three companies. All the more reason to add another party to the mix: YOU!

While the integration is still a bit rough around the edges, we're inviting you to get involved and take the integration in the directions that you'd like to see. We're basing the initial integration on the new CMIS (Content Management Interoperability Services) standard, which means that not only can we build a range of new functionalities on top of the baseline integration, but we can also extend the integration to a diverse range of different technologies.


We've Accomplished So Much Already

The current Alfresco Plugin for Confluence provides a set of custom macros that enable Confluence wiki pages to display or embed Alfresco documents or document metadata through ID reference, Path reference or CMIS Search Query. The expectation is that we can provide a seamless experience to users of both platforms:

• Search integration within Alfresco: By adding OpenSearch capabilities to Confluence, Alfresco is now able to aggregate search results from Confluence wiki pages and the Alfresco repository. So, for example, a user logged into Alfresco will be able to retrieve data from documents hosted in an Alfresco repository and any Confluence page she's got access to. This is a good example of what open standards and extensible systems can offer.


• New macros for Confluence, providing wiki users with the ability to interact with an Alfresco repository in a number of ways, such as:
  
  
  
  • Browse an Alfresco repository
  
  
  • Reference (link) and embed (display directly) documents stored in Alfresco
  
  
  • Build custom reports (such as listing documents that match specified criteria) by running queries against the Alfresco repositories
  
  
  


• Use Web Scripts or CMIS: As an added bonus, macros are implemented using either Alfresco-specific technologies such as Web Scripts or a pure standard-based approach (CMIS). The Web Scripts technology is very, very cool and opens the door to do some interesting new capabilities for Confluence.

This is only the beginning

Even as I type this, an email has hit my inbox noting that new features, like the ability to pull Alfresco Share (Alfresco's collaboration application) site activities into Confluence via an Atom feed, have been added. We'd love to have you help us figure out where to take this integration, and to fill in pieces you think are missing. That's the benefit of open standards and open source: Confluence can be even more than Atlassian envisions.

We're hosting the project on Google Code: http://code.google.com/p/confluence-alfresco/. Please stop by and get involved.


UPDATE
You can also get SSO w/ Crowd for your Alfresco install:
http://code.google.com/p/alfresco-crowd-security/

In a previous post I described how Hamcrest can save your soul. After writing that, it was pointed out that you probably don't need to suffer so much boiler-plate to save your soul.

With that thought I set out to write the deeplyIsEqual matcher. The result is the DeepIsEqual Matcher. Using it is pretty easy. In that previous post I described the Matcher for comparing lightsabers, LightsaberIsEqual. We no longer need that. Now we can simply do

    assertThat(anakinsLightsaber, is(deeplyEqualTo(lukesFirstLightsaber));

No extra boilerplate is needed and we still get all the benefits of only seeing the fields that didn't match in the error messages. Oh, and the reported expected value is built using reflection too, so you don't need to worry about whether the type of objects you're comparing implement toString or how they implement it.

After going through the initial implementation I started switching over the OAuth tests to use this new matcher. Everything was going great until I hit a case where PublicKey or PrivateKey objects were being compared. The problem was, one was a generated key and the other was converted from an encoded string value. Apparently, depending on which way you create Key objects, the internal fields can be slightly different. So, comparing them recursively was failing. I struggled with what to do - I could add the Key to the internal, hardcoded types that are compared by simply using equals(), or I could add the ability for testers to specify how they wanted certain types to be compared using a MatcherFactory. This being 20% time and me seeing other places where specifying custom matchers for certain types would be very useful, I went with the second option.

So, if we wanted to match Hilts in a very specific way we could that pretty simply.

    public static Matcher<? super Lightsaber> equalTo(Lightsaber lightsaber)
    {
        return deeplyEqualTo(lightsaber, extraTypeMatchers);
    }

I wrapped the deeplyEqualTo call in a convience method, because the creation of the extraTypeMatchers can be a bit gnarly and makes the tests a bit harder to read. As a simple example, let's say we want to compare {{Hilt}}s using their equals() and don't care if the subtypes are different. To accomplish that we could use the following extraTypeMatchers.

import static com.atlassian.hamcrest.ClassMatchers.isAssignableTo;
import static com.atlassian.hamcrest.DeepIsEqual.deeplyEqualTo;
import static com.atlassian.hamcrest.MatcherFactories.isEqual;
    Map<Matcher<Class<?>>, MatcherFactory> extraTypeMatchers = new HashMap<Matcher<Class<?>>, MatcherFactory>()
    {{
        put(isAssignableTo(Hilt.class), isEqual());
    }};

That's not very pretty, but by wrapping it up our test can go back to simply being

    assertThat(anakinsLightsaber, is(equalTo(lukesFirstLightsaber));

and we get all the benefits we had before, plus we are matching Hilts in the desired way.

The biggest thing left to tackle is how to handle object graphs that have cycles in them. We have some ideas about how to do it, such as tracking the objects visited and if we find one that we've already visited just stop recursion and assume it will be true, relying on all the other matching to prove otherwise. I'm not 100% sure this will work. The main problem being, what happens if a field in expected value has a reference to, for example, the first object in the graph. But, the actual value doesn't have a cyclic reference but does have a reference to an object that should be considered "equal" to the same object as the expected value (I know, it's probably not entirely clear what I mean but I'm having a problem figuring out how to phrase it more better as my brain keeps breaking when I think too hard about it). I'm thinking for now, being able to specify custom matchers for types should be enough. If there is some part of the object graph that we expect to have a cycle, then just specify a custom matcher for it and move on.

To play with this you can checkout the code from the labs project, or you can get from our public maven repository. If you find any problems with it report them in JIRA.

Thanks, and enjoy testing again!

For Confluence 2.10 (remember that?), we converted the display of the Jira Issues Macro from using a static HTML table to using a table infused with jQuery goodness. Now we could add features that wouldn't have been possible without JavaScript, like the ability to sort issues in the page without even reloading. That was pretty cool, but it also meant we had a new problem to deal with: macros can be rendered in places that can't render JavaScript, such as in a feed reader or an email notification. In those cases, our beautifully redesigned macro would look something like a puddle of goo.

We thought about how to get around this new problem, and decided the best approach would be to make it possible for macros to find out if they are being rendered in an email or a feed, so they can display themselves appropriately. It was already possible for macros to find out if they are being rendered in a PDF document or several other contexts. In Confluence 2.10, we made it possible for macros to find out that they were being displayed in an email or feed, an addition to the previously defined contexts. Previously, macros being viewed in an email or feed reader would have just had the render type "display", which is the default.

Now the Jira Issues Macro is able to render itself differently in display versus feed modes:

Okay, so how can you find out the current render context from within your macro? When creating a plugin that includes a macro module, you return the HTML that the macro will display from the execute() method of the macro class. One of the parameters to the execute() method, the one with type RenderContext, can be used to determine how the macro is being rendered.

Here's a sample execute method from a macro that prints out the current render context type:

public String execute(Map parameters, String body, RenderContext renderContext)
{
 if(RenderContext.FEED.equals(renderContext.getOutputType()))
 return "FEED render type";
 else if(RenderContext.EMAIL.equals(renderContext.getOutputType()))
 return "EMAIL render type";
 else if(RenderContext.HTML_EXPORT.equals(renderContext.getOutputType()))
 return "HTML_EXPORT render type";
 else if(RenderContext.PREVIEW.equals(renderContext.getOutputType()))
 return "PREVIEW render type";
 else if(RenderContext.DISPLAY.equals(renderContext.getOutputType()))
 return "DISPLAY render type";
 else if(RenderContext.PDF.equals(renderContext.getOutputType()))
 return "PDF render type";
 else if(RenderContext.WORD.equals(renderContext.getOutputType()))
 return "WORD render type";
 else
 return "some other render type";
}

If you used this sample macro on a page you were editing (by first installing the plugin that contains it), you could visit the preview tab to see it output "PREVIEW render type". In the case of a more complex macro, you could, say, disable some UI elements when the RenderContext.PREVIEW.equals(renderContext.getOutputType()) check is true. Using these checks is exactly how the Jira Issues Macro decides whether to render itself using JavaScript or just stick with a basic HTML version.