Recent posts from Chris Mountford

First, my definition of refactoring:

Refactoring is improving code without changing the features it implements.

That's all.

If you're refactoring, you're not fixing bugs, you're not improving performance and you not increasing robustness. Refactoring is simply improving the design of the code, while ensuring that it still works the same, warts and all.

Pointy haired bosses the world over froth at the mouth to hear such things. Veins pop out in their temples. You mean the business value of the software stays the same but the cost to the business goes up? I didn't say that.

If you measure business value only in terms of features you have today, then you can end up deep in technical debt; you can add features today in such a way that features tomorrow cost more and more. The value of refactoring is wholly contained in the future ability of programmers to comprehend and modify the code. It's called maintainability, but that's a boring word, so let's call it Agility. Mmmm, sexy. Well-factored code is agile code because it's better able to change.

In economic terms, refactoring is an investment, or the repayment of a debt. It's only worth doing over a time frame when the interest payments or repayments (in the form of ongoing productivity gains) compound to exceed the time invested. Fingers crossed the business or project sponsor is also planning over such time frames.

The term refactoring comes from mathematics. You may remember your high school algebra:

2x² + 10x

Stay with me! No glazing over! If you refactor the expression, extracting the common factor, 2x, you get:

2x . (x+5)

Sometimes it's hard to spot the common factors, in both mathematics and programming, and it can certainly be done poorly. More on that later.

Like most powerful techniques in software development, the purpose of refactoring is controlling complexity.

Complexity is bad, mmmkay. Complexity is evil.

I was fortunate enough to be chatting about project complexity with the legendary Dave Thomas (OTI, Eclipse) at JAOO Sydney in May. He nailed it: "kLOC kills". Complexity and scale in codebases is a major contributor to schedule blowouts, poor velocity and excessive development cost. Complexity is a kitten killer from way back.

Fred Brooks discusses two categories of complexity. Essential complexity and accidental complexity.

Essential complexity is the complexity of the domain. In NASA software, there's no escaping rocket science. You can isolate and divide essential complexity but you can never remove it. Essential complexity belongs to the problem. By contrast, accidental complexity is an artefact of the systems, languages, frameworks you're using. In principle it can be reduced by changing the system. Accidental complexity belongs to the solution. Refactoring reduces accidental complexity.

If you don't have much experience with it and you're looking for some concrete tutorials on refactoring, I suggest you start with Martin Fowler's seminal book Refactoring. Fowler also maintains a catalog of refactoring recipes with an Object Oriented flavour.

One of the most basic techniques is Extract Method which all decent IDEs can do automatically. You know you need Extract Method if you have a multi-page method with a sequence of of comment blocks which look like this: Now that we have the InductionActuator, look up the FluxCapacitor.... Doing it manually means snipping out a logical sequence of code and pasting it into a new, small, well-named method, stitching the local variables used from the originating context into parameters to the method. If this is hard due to sloppy scoping or too many variables, you may consider Introducing a Field from a local variable.

The inner loop of agile development should go like this: Red, Green, Refactor. Red means you have a test which is not passing. Getting the test to pass is the next step. Green means you are passing all tests. Refactor means... refactor.

Even if you're a good agile developer, doing things as simply as possible, complexity and duplication of common factors creeps in while you're trying to pass tests. Everybody hacks. Everyone copies and pastes. This is fine as long as you go back and refactor when you've got the green bar. Sometimes you may need to avoid mentioning this to PHBs, for their own good. Shhh!

Unit tests are really important for refactoring. If you're not doing unit testing you've got a long way to go. A good unit test suite is a necessary precondition for confident, aggressive refactoring. And IMHO a good type system is a necessary precondition for confident, aggressive, automated refactoring. These preconditions can present a quandary for some developers. Legacy systems often have no effective automated tests. And since they're often composed entirely of spaghetti, they need to be refactored. It's a chicken and egg situation, where do you start? All I can say here is you start small.

Refactoritis

Can you have too much refactoring? Absolutely. If you're somewhere around middle-stage zealotry for this refactoring stuff, you may not be in danger of copy+pasting your way to a big ball of mud, but you may fall prone to exceed the safe working abstraction load of your language or go too far beyond the idioms of your team's codebase or comfort.

Every language has limits imposed by its design and implementation. In Java and C#, for example, the limits are seen by many in the dynamic languages camps to be too much to bear. For example, say you're refactoring some Java or C# code. You might create a new interface with a few alternate concrete implementations and, whereas before you had two methods on a concrete class and a few big if-else blocks, after refactoring you might have three files and more actual lines of source code. It can be somewhat subjective but sometimes you may have more complexity even though you've removed duplication!

If this happens you have fallen asleep on the refactoring train and missed your station. Often you should just roll back the code and go write a feature. Some duplication is easy to see and cope with, especially if it can fit on one screen and any reader can see the pattern. In other languages, Lisp comes to mind, there are constructs (like macros) which allow you to encapsulate expressions that cannot be elegantly factored in, say, Java. Disclaimer: IANALN; I Am Not A Lisp Nerd.

So the expressiveness of the language can constrain refactorability. Another way of saying this is that the language contains accidental complexity and only factoring out the language can remove that complexity. I should say here that I have recently found Groovy to be a great candidate for doing this on Java projects.

As a more concrete example, lexical closures are a great way of implementing things like the new for loop (for each) introduced in Java 1.5 and functor frameworks that employ anonymous inner classes in Java for similar purposes (e.g. composable transformers, ad hoc iterator delegators instead of explicit looping) often feel too cumbersome compared to most closure implementations. So you just have to suffer the duplication and code bulk.

So in summary, Red, Green, Refactor, don't go overboard and be aware when your language makes capturing factors you see in your system worse. Kill complexity before it kills you.

If you're interested I'll be telling war stories and going into some side issues over on my personal blog.

Pair programming is easily the most controversial agile development practice and in my experience the most frequently avoided. It makes some people pretty uncomfortable.

Just so we're all on the same page, pair programming means two programmers working together on one computer. Yes really. Normally one codes while the other thinks ahead and around. Or, one drives and the other navigates, swapping regularly. The motivation is higher quality output from the outset.

Pair programming requires working much closer with team mates than most developers are accustomed to. This is both its power and it's greatest weakness.

Working so closely with others can bring otherwise hidden personal issues to the surface. My experience in previous jobs has shown me that pair programming can be so personally challenging for some people that the practice is infeasible for that team. Doing it takes courage and maturity, it requires a healthy environment where mutual respect, humour and humility are all expected. Not to mention minimum standards of personal hygiene! This is quite different from the stand-offish formalism characteristic of many alternatives and this humanistic style sets the bar too high for some. Cowboys, incompetent introverts, control freaks and superhero programmers all resist pair programming. It's like their kryptonite.

Like culture shocked westerners first visiting Tokyo, unused to the dramatic reduction in personal space one can expect, say, on a commuter train, for some, getting used to the personal invasion that is pair programming is a self-imposed barrier they just need to break through.

Pairing up with different people makes this personal contact a dynamic challenge. There are several axes of individual difference which demand different behaviour. For example there's the novice-expert axis and the introvert-extrovert axis and people have different natural working styles and favoured problem solving strategies. For example I can often be willing to explore unconventional solution paths which make methodical people a little unsettled. Being aware of that helps me communicate appropriately and change my behaviour where necessary.

Pairing requires bilateral adaptability and the art of compromise.

There are many common anti-patterns that lead to ineffective pair programming. These things are fairly well documented, for example see Pair Programming Illuminated by Laurie Williams and Robert Kessler. It's worth tackling these problems actively because ineffective pair programming is worse than no pair programming and it's obvious even to would-be starry-eyed XP fans.

One of the reasons I've been thinking a lot about pair programming recently is because I haven't been doing it for a change. (gasp!)

Apart from some time on support and fixing a pile of trivial bugs (not pairing), I've been working solo on a front-end feature for JIRA 4.0. Solo because our team has an odd number of people right now (wanna job?). It has actually put me in a great position to evaluate the practice of pair programming somewhat objectively.

First the good stuff about not pairing. I have some great new headphones and with the right music I can get into the zone and stay there for hours without needing to coordinate with anyone. I do not have to continually exert effort listening or thinking of ways to express my questions or ideas. I just think and do. That's refreshing.

But I'm not ashamed to admit it's a little isolating. Hey, I'm a people person.

The negatives of not pair programming are especially visible when you are accustomed to doing it. While not pairing recently I have become curiously suspicious of the quality of my work.

I think it's good, Is it really good? I'm not gold-plating am I? Maybe this is absolutely the wrong approach. Is there something obvious that I'm missing? I'd better get someone to review this.

All those thoughts are plaguing me in my "step back moments" when the tests pass and I get to wonder about the next move. This wouldn't happen if I was pairing. I would be in dialogue with my partner; we would have common mind (just imagine Yoda saying that last bit).

Pair programming can really help in making a team gel. Spending time together - not just in pair programming but in other activities like sharing a meal or social event during work time help teams to gel.

So while there are many reasons to practice pair programming, it's not the only way to get these benefits. Code review is an obvious alternative for many of the same benefits. Marketing may want to break my arms if I don't mention Crucible at this point.

So if you're not improving the quality of your code with pair programming, then I'd hope you are doing code reviews because there really isn't much disagreement in the industry that this is a very well known and effective method of ensuring quality software development.

Atlassian development teams do either one or the other, or sometimes, like in JIRA some of both.

Using a tool like Crucible for code reviews has some advantages over pair programming, such as record-keeping and working across distributed teams and time zones. Pair programming has some advantages over code review such as swapping keyboard shortcuts and shell one-liners and helping a team to gel.

So if you have a business case for better quality software then remember: two heads are better than one!

Learn about agile software development inside Atlassian

In 2007 I did a series of surprisingly popular blog posts on Atlassian's Agile Process. As part of our renewed push to share our own agile story, I wanted to post an update to my earlier posts. Last time I enumerated the XP practices and mapped them to Atlassian's. I won't be repeating myself (D.R.Y.), so if you're interested in the origins of Atlassian's Agile Process, here are parts one, two, three and four.

Today in the Engineering division, we still have an agile process in each of the development teams defined for and by that team, that is as true now as it was in 2007.

The things that have changed are mostly in the category of second order effects.

Proliferation

The biggest difference in Atlassian's agile process in the last two years is that now it's not just a process for engineering work. Every department is getting agility. Top books on Agile practices are handed around, stacked available on common bookshelves and regularly seen on the tops of busy desks, scruffy yellow sticky notes protruding.

And now I'm even beginning to think that people are reading them.

Another sign of mass agile process adoption was about a year ago when middle management (which was at that time brand-new at Atlassian) started doing stand-ups. After that it seemed as though a critical mass had been reached and the practice of stand-up meetings became ubiquitous. It seems there is a culture of minimalism in meetings, not just at Atlassian, but amongst all smart people who want to get things done.

It's not just Stand-up Meetings either. User stories, You Aren't Going to Need It, Do The Simplest Thing That Could Possibly Work, Don't Repeat Yourself as well as some of the foundation thinking such as kaizen are all now part of the Atlassian lexicon.

I attribute this adoption to three things:

1. People have seen it work.
2. Leaders like it.
3. This stuff isn't new-fangled, it fits in with much established wisdom.

Scaling

Atlassian has grown quickly over its six year history, both in staff and in the number of offices. Processes have had to scale and over long distances and time zones. A good example of the growing pains is JIRA's automated test suite. It's an awesome beast. There are a growing number of unit tests (junit), functional tests (jwebunit) , web browser driver tests (selenium) and there is now a QA department who reviews and improves all aspects of our quality assurance regime which includes testing.

JIRA supports multiple operating systems, JDKs, application servers, databases, and versions thereof. And it comes in three editions (Standard, Professional and Enterprise) with different feature sets. Each edition comes in different distributions including EAR/WAR, Standalone which includes Tomcat as an application server and there is a Windows installer executable. As if that wasn't enough, it works in multiple versions of different web browsers.

So that's a lot of combinations to test! We really wish we could automatically test every combination! Currently our full functional test suite takes more than two hours to run. Multiply that by every combination and it would probably take several days of CPU time to complete. Of course Unit Tests run comparatively quickly and their green bar gives fairly good confidence, but if we want to be sure we haven't broken something in the JIRA user interface when we refactor the code, we have to wait at least two hours.

Running many thousands of functional tests is what is known as an embarrasingly parallel workload and accordingly, I am embarrassed to say we have not completed the work to properly "parallelify" it.

But now that Bamboo supports agents running in the EC2 cloud, we can crank up the compute resources available for continuous integration and we should be able to bring our functional test latency down under 20 mins without too much trouble. Fingers crossed.

Code Review

"Pair programming is the ultimate code review" goes the "agilist" line. I can see this. I can see how it works and I have felt code-review benefits from pairing. I'll be blogging about pair programming specifically in the coming weeks, so I won't exhaust this side of the argument here, however I have to say that code review is the single most effective quality assurance measure on top of a traditional agile process that I know.

There are many advantages of code review over pairing as well. Of course, these are not reasons to abandon pairing, because pairing provides much more than code review. If you are using a web-based code review tool like, ahem, oh look here, it seems we have a product I can pimp here... Crucible... the main advantages are the most obvious ones. You can scale up the number of reviewers and they can act from different locations and time zones. You also get a permanent record.

The JIRA team is probably an average adopter of code review as a practice where it remains a discretionary thing. This suits us because it lets us always use the Agile advised dosage: Just Enough.

Virtual Index Cards with GreenHopper

While we really like the simplicity of paper cards for tracking tasks in the small, the electronic version can be better in some respects. Not only for distributed teams. Of course we use JIRA for tracking large numbers of bugs and feature requests, but it was not until the GreenHopper plug-in that we have felt that JIRA was fully capable of giving us the immediate feedback and visibility that cards can give a co-located team. It also makes burndown charts which are increasingly to be found on big LCD screens around the offices.

So that's it. We like our agile process. It suits Atlassian.

With a title like that you may expect me to say the uncertainty principle is the fact that nobody knows how long it will take to develop a piece of software. Well it's not, it's quantum mechanics, man, and you're never going to understand it.

I may not understand it either but I can still blog about it!

Apparently, the quantum mechanics view of the world is that of the possible alternatives, each unknown future exists simultaneously until an act of measurement is made and the matter is decided, whereupon this cloud of possible futures collapse into the "actual" one that was observed. WTF? Yeah I know.

Usually this kind of craziness is discussed at the subatomic level where our standards for common sense are much lower. If you're completely lost, you should go read about Schrodingers Cat.

So what has that got to do with software?

Here's an interesting tweet from Jason Yip on productivity without testability:

What does it even mean to have high developer productivity but very low testability?

And I thought "Good question". But I had no witty response to clog up twitters servers with and so left it to percolate in my mind.

Then I realised is that unit testing is a measurement which decides whether or not a feature is present. If the test fails, the feature is absent.

Untested features exist in a quantum state where each of the possible states of implementation are simultaneously true until the point at which a test is written (or a user tries to use the feature) and the matter is decided.

So untested features may exist, or they may not, but you won't know until you write the test.

Just thought I'd document an important recent scientific finding of mine.

Code about which somebody says "I've already got it running on my laptop" will, when moved to a work machine, be found to not function.

There is clearly some kind of relativistic effect going on here which is specific to the frame-of-reference of one's own laptop.

This is the general law.

The special law of laptop relativity is that if the code was written on public transport, it will be found, from the frame of reference of the work computer that the code "could never have worked".

While the Atlassian User Groups have been popping up all around the globe, here at Atlassian Headquarters in Sydney we have yet to host an AUG. That is until now.

In a little over a month, in time for a much awaited southern hemisphere springtime, Atlassian's Sydney offices will host the inaugural Sydney Atlassian User Group meeting.

The event is open to all users of Atlassian products, interested parties and partners.

Being on home turf, this is sure to be well attended and we want to make it something special. If you're in Sydney or able to get to Sydney in September put the AUG meeting in your diary.

The programme is sketched out on the event page and includes talks by Atlassian CEOs Mike Cannon-Brookes and Scott Farquhar and real customers talking about their experiences with different products.

If you're interested in participating or have some ideas or perhaps something to present in one of the birds-of-a-feather sessions, get in touch.

Links to further details and RSVP are on the Sydney Atlassian User Group event page.

While I have been long familiar with Design Patterns in software development and find myself using a reasonably broad pattern language in my daily work, I had not until recently been made aware of the Simpleton Pattern.

I shared my new found knowledge with my colleagues and found great enthusiasm for it and their hopes for its proper inclusion in the literature. A quick consultation with Google and I see that there is a significant body of knowledge about Simpleton.

Nevertheless it has proven hard to find agreement on its specific characteristics.

If you do your own research you may be led to believe that Simpleton is an anti-pattern. It has also been noted that Simpleton is best suited to the BBM architectural style. Some engineers at Atlassian maintain that the Simpleton characterises properties of developers themselves rather than of the software.

At first it may seem that implementing Simpleton requires great skill and attention to detail. In fact the opposite is true! Simpleton doesn't require special training or framework support.

In fact if you look at some of your own legacy code you may be surprised to see that you have in fact unwittingly produced several implementations yourself. I know I have!

So I make this post in the hope that as a community we can grow our understanding of this important pattern. Next time you review some code check for the signs. You'll know it when you see it.

Everything should be made as simple as possible, but not simpler. - Albert Einstein

With the current trend in unconferences, it seems people have finally become tired of all the marketing hype that drives so many traditional tech conferences. Just thinking about it is giving me flashbacks of sitting in a darkened auditorium for the last session of the day fighting off the boredom as people around me begin drooling out the side of their mouths, dead eyes unfocused on the conspicuously excited spotlit suit and a seemingly endless series of point-form slides and graphs with unlabelled axes. Reminding myself there were crates of beer in ice being deployed out in the hall was the only thing that kept me compos mentis.

Of course, not all conferences are only as good as the parties after hours. And I'm sure I speak for several others when I suggest that it's a great place to start.

But there seems to be a niche for conferences that don't suck. A conference by developers, for developers.

When we heard there was a series of conferences coming up that aren't full of marketing fluff, we were interested in Atlassian being involved. When we heard JetBrains (most Atlassian developers worship at the temple of IntelliJ IDEA), Contegix (these guys do all our hosting) and Jive (we have run the awesome Jive forums for years) were sponsors we knew No Fluff Just Stuff was something Atlassian should be in on.

Of course, when I heard about it I was ready to be disappointed. Being a chronic sceptic my thoughts about the NFJS conference were as follows:

• What the hell is that? The name doesn't tell me anything.
• Oh I see it's a conference - yawn.
• "No Fluff" eh? I bet that's marketing trying to convince me of something I'm least likely to conclude from the evidence - it must be totally full of fluff.
• I wonder if the parties are any good.
• Atlassian are "proud partners" with NFJS. Why are partners always "proud"? Why can't they be "splendid" or "glad" or "titillated" just for once?
• Laura from marketing tells me "it's actually good" and I should check the website for myself.
• Oh alright then.
• Hmm... I just read the blurb and my bullshit detector is not going off.
• OK maybe it doesn't suck.
• Damn. Wish I was in Boston this week.

So the No Fluff Just Stuff conference does seem to be a pretty good Java developer event. They hold it in a different US city every week and I have heard talk of NFJS doing some conferences outside the US maybe next year.

At the actual NFJS conferences we've been giving away free licenses to the shiny new JIRA Studio so if you're a free-stuff person thinking of attending, go and bug our people to find out how to win one.

Hopefully NFJS is the part of a broader movement and more conferences that don't suck appear.

Start at our titillated partnership page and watch for further updates on the news blog

In this final part of my series on Atlassian Agile Process I wrap up and reveal XP's fundamental weakness. Oh and I cover socoipaths! And Ninjas. But no pirates. See also Part 1, Part 2 and Part 3.

Some Other XP Practices

Let me get the rest of XP out of the way. Sustainable Work Hours, Collective Code Ownership and Metaphor are XP practices we don't give much thought to. The first two we embrace completely and the third we pretty much ignore. Then again, Bamboo has a metaphor: Telemetry. Perhaps Metaphor is more important towards the beginning of a system's life.

Extra Practices

The following essential practices for our process are not defined in XP:

Specifications

Purists and detractors mistakenly think specifications are in conflict with XP ideals. We aim for our specifications to be both minimal and complete. Which is a blend of just enough documentation and turning the knobs to eleven (for that extra push over the cliff). They are written in Confluence to a template with a checklist for the dozens of essential aspects to a new feature. The checklist includes things like security, user interface design, caching, remote APIs etc. From the spec comes a stack of task cards.

Training

Alternating fortnightly with the process improvement meeting I mentioned in Part 1 the JIRA team has an hour for internal training. Confluence has a similar regime. Cross-team involvement is growing. In addition to team-internal technical talks, Atlassian has recently started larger cross-team developer training events. For example last week we were treated to one on GWT by the dynamic duo, Bruce Johnson and Joel Webber who were in town at Google Sydney. Increasingly training goes on outside the software process.

Hiring

Hiring, I think, should be a specified part of any development process. It's something I've been involved with for a long time and something I've learned about through making mistakes. Hiring also has a huge impact on your process.

Our agile process requires people to spend the effort to listen and talk to each other, working closely. You have to be a people person to like it. It doesn't suit sociopaths. Accidently hiring a sociopath is going to make XP impossible. Trust me, I know. To me this is XP's fundamental weakness. Of course if you only hire ninjas, you probably don't need much formal process. Ninjas just do what needs to be done.

Hiring great people is something we take very seriously at Atlassian and we try to make it quick and effective. Our interviewers are busy developers and our favourite candidates are going to get other offers. The hiring process involves screening, phone and face to face technical interviews and a coding test. We are conservative. We would rather err on the side of not hiring someone. Of course the problem is Ninjas are very hard to find.

We also do a lot of things to keep people happy once we have hired them. We have an awesome work environment and excellent tools. We are also hiring right now! Sociopaths need not apply.

Last week I posted part 2 Atlassian's Agile software development process. Part 1 was the week before. This week I deliver the next iteration. Update: part 4 now available.

Continuous Integration

Continuously building and automatically testing our software is essential for quality. We have many dozens of combinations of editions, distribution types and different versions of supported servers and JDKs on the subversion trunk as well as the stable branch. That's for each product. We need continuous integration servers with muchos grunt. Just for JIRA (not including plugins) we have a high end four way continuous integration server (and a second box on order). We have prominent displays advertising the health of our builds and tests (green is good, red is bad) and we take every opportunity to laugh and point at other teams whose builds may be currently failing. Of course we use Bamboo for continuous integration! CI Newbies should start with Martin Fowler's canonical introduction.

Small Releases

Iterations are one week long, releases are 6-12 weeks long, depending on the product and the scope of the release. While some customers like high release frequency, others have aversion to the perceived upgrade overhead that our releases push onto them. Some customers are not comfortable receiving notification of a new release too frequently, nor of falling too far behind the latest available version. Releasing less often is an option, however we would still fight to ensure that we produce production-worthy software at the end of each iteration.

Making the software available for download often seems to engineers as an undeniably good thing. Low latency is good, right? If you don't want it just don't upgrade, right? Well it's more complex than that. This topic is on high rotation for debate at Atlassian.

On Site Customer

We have thousands of customers, but everyone in the company is also a customer of our software. We self host. We eat our own dog food. In fact Atlassian lives on its own dog food. Dog food is yummy. It's critical to understanding how the software should be improved.

Nevertheless we sometimes struggle to see things from a customer's unique point of view. We know how we use our software but people use our products in all sorts of crazy situations beyond our original expectations. So, having recently filled specialist roles, we are growing coherent product management. These folks will become our customers on site.

Refactoring

Refactoring is about not getting into technical debt (or slowly getting out of it as the case may be). Like with many of our practices we have practice champions whose job it is to monitor and report on team performance for a specific practice. For example the JIRA refactoring champion is currently Dushan, a black belt in karate. Mostly the team accepts that checkins of, say, copy and pasted code could put us in hospital. It's tough but fair.

Next time...

In part 4 I plan to cover the remaining XP practices and also two practices we think are essential to Atlassian's Agile process that are not part of XP. Unfortunately, because the topic of sociopaths is the lowest priority item in the backlog for this series of posts, I'm going to have to push it off until part 4. I have to cut scope, I'm at the end of my time box and I have to ship what I've got. Sorry about that. Next time for sure ;)