Recent posts from Don Brown

Plugins are arguably the killer feature for Atlassian products, as they allow you to tweak a theme or deploy full-blown applications within a familiar environment and infrastructure. The number of plugins available, especially for established products like Confluence and JIRA, is huge and the amount of extension points available to plugins basically give you full control of the host application. That much power comes with a price - your plugin is heavily tied to the product, and anyone who has done something similar, like tried to write an application on top of Microsoft Excel with a ton of VBScript, knows how sensitive your Frankenstein application is to host application changes and upgrades, let alone bugs and stability issues.

In JIRA Studio, we are working with the common platform of JIRA, Confluence, Fisheye/Crucible, Crowd, and Subversion, but the applications the user will visually interact with are JIRA, Confluence, and Fisheye/Crucible. If we want to add a new feature like, say, a top navigation bar, we have to add it to each product as a plugin. Since plugins are tightly coupled to each product's APIs, we have to basically write three different plugins. Here are a few of the differences in the latest version of Atlassian products we have encountered so far:

As you can see, when writing even a moderately complex plugin, you have to be very aware of the differences across the applications, even in fundamental areas like logging. Furthermore, the differences will force you to write three plugins instead of one: three times the code, unit tests, functional tests, and potential for bugs.

Creating the Studio Application Access Layer (SAAL)

For our own sanity, we've created the Studio Application Access Layer (SAAL), which is a set of service provider interfaces for infrastructure-level features that allow a plugin to rely on key services provided by a single API without having to deal with the multiple implementations. SAAL includes interfaces for the following features:

• Service object lookup
• Logging
• Settings persistence
• HTTP calls
• Internationalised message passing
• Plugin upgrade framework

Our thought is to grow this layer organically for our needs when writing plugins for JIRA Studio. This allows us to a) minimize the amount of code to maintain and b) minimize the conceptual surface area for plugin development. The end game is to have the code interfaces and classes put into Atlassian Plugins, with the individual products responsible for the product-specific service implementations.

This is a good example of how creating JIRA Studio is actually helping the core products as much as creating a new one. You'd think that every product at Atlassian would follow the same development practices, use the same libraries, or even implement individual features the same way if developed at the same time, but alas, that is not the case. Different teams have different leads with different developers, which all amounts to slightly different interpretations of Atlassian ideals, and of course it isn't always feasible or even advisable to, for example, spend months rewriting your web layer with no customer benefit because WebWork 2 was released. A secondary goal of JIRA Studio is to confront those technical differences and try to wrangle them into alignment for the benefit of the customers of both the Atlassian product and JIRA Studio.

For the last few months, we've been hard at work creating the newest member of the Atlassian family, JIRA Studio. Since this product is targeted squarely at developers, we decided it would be useful to engage the developer community to talk about what we are working on, both the good and the bad.

Our team uses a modified XP process that splits work into one or two week iterations, recording tasks using both JIRA and the more traditional note cards. In this last iteration, numbered 114, our primary focus was to create a "FireBall"™ (our product manager makes us call it that), which is basically a tarball containing:

• Confluence, JIRA, Fisheye/Crucible, and Crowd WARs
• Tokenized backups of each application.
• An installation script that reads a properties file and modifies the backups for the specific instance to be installed (changing the hostname, mail server, etc)
• README.txt detailing the steps to install the FireBall™

Our plan is to deliver this FireBall™ to Contegix, our hosting provider, for every new version. The system that creates this FireBall™ is our integration test system, since it needs to deploy the whole JIRA Studio set of applications and run our integration tests against it. Granted, we use different backups for the tests, but the process is fundamentally the same. This means that every time Bamboo runs our integration tests, usually after every commit, we not only get a full integration testing of JIRA Studio, but create the deployable FireBall™. Using the same system for release artifacts and testing helps ensure our release process is maintained and regularly exercised.

Speaking of functional tests, in Iteration 114, we went on a testing spree, writing heaps of tests for our project creation feature. In JIRA Studio, when you create a JIRA project, we create a:

• Confluence space with the same key and name
• Subversion directory tree (trunk/tags/branches)
• Fisheye repository for the new Subversion directory
• Crucible project with the same key

Obviously, there is a lot that can go wrong here, so we spent a week writing function tests to cover all the error conditions we could think of. Writing functional tests that cover so many products is tricky (and the performance of Maven 2 doesn't help), but the benefit far outweighs the cost.

Finally, another core feature of JIRA Studio is the navigation bar that sits above each application. The static images and Javascript currently comes from a directory, served by Apache, mounted at "/static". To remove that step from installation, we wanted to bundle the static resources into each application. JIRA and Confluence have a plugin type called a "web resource", that allows you to define static resources to be made available to the page. It exposes the resource using a special unique URI that enables the use of the HTTP caching headers that encourage the browser to cache the resource for a year. This dramatically speeds up the application for the user.

Anyways, we moved the static resources into a new plugin that can run unmodified on JIRA and Confluence, which just leaves Fisheye. The latest release of Fisheye/Crucible supports plugins, but didn't have that plugin type available. Taking advantage of the Fisheye/Crucible guys a few desks down, we bribed them to add the web resource plugin type, so we now have a plugin that can run on JIRA, Confluence, and Fisheye/Crucible, simplifying our deployment and speeding up JIRA Studio in the process.

Well, that's enough for now. I hope to keep the blog updated with our progress, and of course, feel free to let us know what you'd like to see in JIRA Studio by commenting to this post or emailing our feedback address: studio-feedback@atlassian.com.

An important lesson in software is to use it for what it does well, and don't try to force it into areas it isn't meant for. Blogging has become an important communication tool for many companies, both outside and inside the firewall. Confluence provides decent support for blogging, and a good match for behind-the-firewall internal blogs that need connectivity to core business systems rather than optimised public access. On the other hand, Confluence isn't as well suited to be a public blog, particularly one that receives a lot of traffic and malicious attention.

At Atlassian, we use Confluence very heavily for internal blogging. Every department, from engineering to sales, uses internal blogs to communicate with the company at large and document internal discussions and announcements. For our external blogs, however, we chose to use Movable Type, a popular and fully-featured blogging software by Six Apart. While clearly a case of the best tool for the job, having two blogging systems can be rather unwieldy, particularly when a rather insightful internal blog post is voted to be promoted to one of the external blogs.

Enter One-Click Blog Publishing

My FedEx day project was to create a Confluence plugin that provided a "Publish" button on top of each internal blog post, allowing the author to easily send the post to our external blog system. While not the most complicated FedEx project, I hope it will be one of the more useful ones by saving us time and ensuring our external blogs are kept up to date.

When the "Publish" button is pressed, this happens:

1. The destination blog and internal post id are sent to an action
2. The action gets the internal post and renders its contents
3. The action also retrieves any post attachments and publishes them as Movable Type resources using their MetaWeblog API
4. The rendered post is processed for link rewriting, using the new link URL's from the newly created Movable Type resources
5. The rendered post is sent as a draft to the Movable Type blog, again via their MetaWeblog API
6. Finally, the user is redirected to the "Edit Draft" page, so they can tweak the presentation and publish

While this plugin is meant to work with our Movable Type blog, any blogging system that supports the MetaWeblog API should be compatible.

What's Missing

In a nutshell, any pages for configuration. Before this went live, I would expect to see:

• A screen to add external blog servers
• A screen to allow users to map their account to their Movable Type account
• Content massaging to preserve the look and feel from the Confluence post

Conclusion

Within a company, this plugin can enable a very useful publishing workflow that gets other people involved in the blogging process. It lets blog authors construct a post and publish it internally, gathering feedback and making improvements as necessary. Since the internal post is also a wiki, multi-author collaboration now becomes easy and second-nature. Once the decision has been made to push the change public, this plugin makes the publishing step a one-click operation. No more emailing word documents around; no more copy/paste from system to system. Now even blogging can be collaborative.

Confluence and JIRA have a great plugin system at their core that allows you to install collections of actions, Spring beans (Confluence), jobs, etc. as discrete plugins. Confluence supports hot-deploying these plugins so they can be added, removed, or upgraded without bringing down the application. This capability is so powerful that internally we are moving more and more functionality into plugins as a way to organise our codebase and keep it tightly-focused and agile.

Wanting this same functionality in Struts 2, but enhanced to support versioned transitive plugin dependencies (plugin A uses services from plugin B which uses services from plugin C) that are automatically resolved and verified at runtime, I wrote the first cut at an OSGi plugin for Struts 2. This plugin allows you to group your actions and services into different jars, namely OSGi bundles.

The goal is for the Struts 2 developer, very little OSGi knowledge will be required and eventually the plugin will include features like a bundle management GUI (also deployed as a Struts 2 bundle) and integration with Spring's OSGi project to make bundle development as easy and close to normal development practices as possible.

From a technical point of view, what I did was create an OsgiConfigurationProvider, which starts an embedded Felix OSGi container and automatically loads bundles it finds in 'WEB-INF/classes/bundles'. It loads the bundle's struts.xml configuration file, inserting packages into the main Struts 2 configuration. The plugin also overrides the ObjectFactory (delegating to a user-defined ObjectFactory) so that classes can be loaded from bundles if not found in the application's classloader. Finally, to enable Velocity resources to be resolved, it adds a Velocity ResourceLoader that also iterates through the bundles looking for the desired template. The basic integration pattern is what is used today in the Confluence source code.

Therefore, from a user perspective, they will need to add a few lines to web.xml to enable the configuration provider:

<filter>

    <filter-name>struts2</filter-name>

    <filter-class>org.apache.struts2.dispatcher.FilterDispatcher</filter-class>

    <init-param>

        <param-name>configProviders</param-name>

        <param-value>org.apache.struts2.osgi.OsgiConfigurationProvider</param-value>

    </init-param>

</filter>

Then, drop the Struts 2 OSGi plugin in your /WEB-INF/lib and the plugin is fully installed.

More information can be found on the plugin's documentation. The plugin is still in the very early stages and as such, isn't ready for production as it hasn't been tested heavily and is missing key features. Still, it, along with Spring's OSGi project, holds a lot of potential to make truely hot-deployable web applications easier for everyone.

While its great that applications such as Confluence keep track of errors by writing them to the logs, they are generally ignored as there is no convenient way to access them. I set out to solve this problem by writing a script (built on Enchanter) that processes log files on a remote server for any error messages, then creates an RSS feed. Here are a couple feeds I created for some servers I was interested in:

The base Enchanter script that generates these feeds is log4j-rss.bsh. The script generates the RSS feed when activated, so to keep the feed timely, you'd probably want to schedule via a cron or similar tool. I use this script, scheduled via cron, to monitor several Atlassian and Apache servers, and I'm sure it could be useful anywhere else, since it isn't tied to the log format of Confluence, Tomcat, or even Log4J.

With the recent press covering big companies adopting OpenID, I decided to see what the fuss was about, and to take it a step further, modify Confluence to be an OpenID consumer. With some time to kill on a Saturday, I finished a fully functional prototype.

OpenID is an open, decentralized, free framework for user-centric digital identity. It basically allows you to log into one application using another for authentication. For example, AOL now supports OpenID, so the 63 million AOL Instant Messenger logins can now be used with any OpenID consumer. I modified Confluence to allow a user to use their OpenID account, an AIM account in my test case, to log into Confluence.

Let's walk through how it works:

Step 1 - Enter your OpenID identifier

At the login screen, I added an OpenID text field for your OpenID identifier. In this screenshot, I'm using my AOL OpenID account:

Step 2 - Login on the OpenID server

When I submitted my OpenID account on the Confluence login page, I was redirected to the login page of my OpenID provider, in this case AOL. I entered my AIM user name and password and clicked "Submit".

Step 3 - Automatically logged into Confluence

I'm automatically redirected back to Confluence and now I'm logged in. Notice in the upper right my name is just my OpenID identifier for now. If my OpenID server supports it, other attributes like my full name, email address, and any thing else could be retrieved from the OpenID server.

Implementation

The implementation in Confluence was pretty straight forward, however, it was difficult working with the OpenID library I chose, OpenID4Java. There really isn't any solid Java library right now as OpenID seems to be currently more active in the Python and Ruby camps. OpenID4Java worked ok, once you get past the zero documentation and outdated code examples in the javadocs.

In Confluence, I created an OpenIdAuthenticator, which extended the usual ConfluenceAuthenticator. Other than the authenticator and related configuration, the only other change was the new OpenID text field on the login form.

Future work

If this code went into production, we'd probably need to spend a day or so cleaning it up, adding better error handling/reporting, and adding a couple of features:

1. The ability to enable or disable OpenID logins in the Global Settings
2. A signup form for new OpenID users when the required information cannot be retrieved from the OpenID server (for example, their email address)

I'd also like to see Confluence as an OpenID server. This has a lot of potential, thanks to the personal space feature of Confluence, because in the ideal case, your identity is tied to "your" home page for that application.

Finally, I'm curious how far you could take OpenID, particularly for companies. We could turn the Atlassian website into an OpenID server, then allow any of our apps, including those hosted elsewhere, the ability to log in using their Atlassian id. The lure of decentralized identity management is certainly powerful, particularly for non-heterogeneous such as the Internet.

I occasionally write Confluence plugins that use nightly jobs to perform long-running tasks. While you can get pretty far with unit testing, sometimes you want to force the job to run while you are clicking around Confluence. To make my job easier, I wrote the Job Manager plugin. This plugin displays the current jobs scheduled by Confluence and any plugins, as well as allowing you to run, pause, or resume execution of these jobs with a click of the mouse.

Be warned - internal Confluence jobs shouldn't be manipulated, particularly on production systems, unless you are absolutely sure you know what you are doing. This plugin is best used on a development instance while testing plugins that involve infrequently run jobs.

Here is what the Job Manager page looks like:

Useful resources:

• Job Manager Documentation
• Download, or you can use the excellent Plugin Repository (recommended)

Tired of keeping your source code comments or code snippets in sync with your documentation? Introducing the Snippet Plugin 2.0, which allows you to include text snippets from external URLs such as Subversion or ViewCVS in your Confluence pages. The plugin supports the ability to define lists of accepted url prefixes to protect against abuse.

For example, in the plugin configuration, you can map the following prefix to URL:

  myapp/   -->  http://www.mycompany.com/svn/myapp/

{snippet:id=description|url=myapp/README.html}

You can easily import the snippet from a file, README.html, within your Subversion source code repository:

...
<!--START SNIPPET: description --!>
This application provides. . . .
<!--END SNIPPET: description --!>
...

Snippet URL's aren't limited to plain or HTML text. You can retrieve snippets from Javadoc comments in Java code, and the macro will know how to strip unnecessary characters. Also, you can refer to Java classes directly and the macro will know how to convert the Java class to a URL. For example, with the appropriate URL prefix defined, the macro will map this Java class to a URL:

  com.mycompany.myapp.MyClass  --> http://www.mycompany.com/svn/myapp/trunk/src/java/com/mycompany/myapp/MyClass.java

See the Snippet Plugin document for more information.

This new version of the Snippet plugin provides:

• Configurable URL prefixes
• Snippet errors collected and managed per-space and globally
• Plugin configuration page
• Support for processing Java and Javadoc classes and snippets, respectively

Useful resources:

• Snippet Documentation
• Download, or you can use the excellent Plugin Repository (recommended)