Confluence has a Selenium Build!
Agnes Ro, Confluence Developer talks about ConfluenceMay 11, 2008 5:15 PM
Last week, Matt Ryall and I have been trying to get a Selenium build up to test Confluence's Javascript features, especially the new drop down menus and page ordering tree. It wasn't easy, but I personally think it is definitely worth having these tests.
Below is a summary of the steps and problems we encountered in setting up the build.
1. Writing a Selenium Test
A few weeks back, I did a Selenium spike to see how easy it was to write a test that logged into Confluence and opened a add drop down menu. Surprisingly, this was very simple to do. I had to add the selenium client dependency in my test module and then used the Selenium Maven Plugin to start the Selenium server.
protected void setUp() throws Exception { super.setUp(); String serverLocation = "localhost"; int serverPort = 4444; String browserStartString = "*firefox"; baseUrl = "http://localhost:8080/confluence"; sel = new DefaultSelenium(serverLocation, serverPort, browserStartString, baseUrl); sel.start(); } public void testNewAddMenu() throws Throwable { logInAsAdmin(); assertEquals("Dashboard - Confluence", sel.getTitle()); gotoPage("/display/" + TESTSPACE_KEY + "/Home"); asserter.assertElementPresent("add-menu-link"); sel.mouseMove("add-menu-link"); assertTrue(sel.isVisible("createPageLink")); assertTrue(sel.isVisible("createNewsLink")); }
2. Running the Selenium Test in Maven
During my spike, I also tried adding the selenium test I wrote above to the integration-test phase in maven. I had to add some config to start and stop the selenium server before running the tests.
<plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>selenium-maven-plugin</artifactId> <version>1.0-beta-2</version> <executions> <execution> <id>start-selenium</id> <phase>pre-integration-test</phase> <goals> <goal>start-server</goal> </goals> <configuration> <background>true</background> <port>${selenium.server.port}</port> </configuration> </execution> <execution> <id>stop-selenium</id> <phase>post-integration-test</phase> <goals> <goal>stop-server</goal> </goals> </execution> </executions> </plugin>
3. Selenium Test Infrastructure
Currently our acceptance tests do not have the best code infrastructure. For example, we have a huge AbstractConfluenceAcceptanceTest class that all the acceptance tests extend and it basically has everything in there. So it was nice to be able to start from scratch and split things out into separate classes. As we wrote more tests, we saw a lot of code duplication and realised how silly the DefaultSelenium client was. We ended up extending the default one to override methods with some sensible implementations. One example is the the client.open(url) method that opens the browser up in the provided url. In a test, you will always need to wait for the page to load before making assertions or clicking/typing on the page.
/**
* Unlike {@link DefaultSelenium#open}, this opens the provided URL relative to the application context path.
* It also waits for the page to load -- a maximum of five seconds before returning.
*/
@Override
public void open(String url)
{
super.open(contextPath + url);
super.waitForPageToLoad(MAX_WAIT_TIME);
}
The Selenium client also doesn't provide any convenient assertion methods. The Jira guys seemed to have a nice assertions class, so we just 'borrowed' it :) Ideally, we will have common code like this in a separate project.
4. Setting up the Bamboo build
To run the Selenium tests in a separate build, we wrote a new pom that only ran Selenium tests after Confluence setup. This was fairly trivial and passed on our local machine.
Up until this point, everything seemed surprisingly easy. Matt & I were starting to really like the Selenium tests... (You can actually see Firefox launched and it flickering through the tests!) However, it was when we actually setup and the build in bamboo that we started to hit a few problems.
Firefox wouldn't start in Bamboo
Since we got the tests running locally, we thought it wouldn't be much more difficult to run on Bamboo. The first problem we encountered when we kicked off the build was that firefox wouldn't launch and therefore not even run the tests. The first error message we got was something about the Selenium server being started on a headless system and was smart enough to suggest using the selenium:xvfb goal (also provided by the Selenium Maven Plugin). So we did as it said and modified the pom to started the Xvfb instance before starting the Selenium server.
After that, we got no other error message, but Firefox still wouldn't start. It would simply fail and eventually timeout. We got to a point where we could ssh into the bamboo box and manually run the maven command to run the tests on the bamboo boxes. Strangely this worked?! We decided to take a look at Jira's build and see what the difference was. This is when we realised that they were running a vncserver before the running the tests! It seems like bamboo doesn't have access to a console and this causing problems when Selenium tries to launch Firefox. As soon as we ran the build with a vncserver, we actually got the tests running - but not quite all passing...
Intermittent Test Failures
The annoying thing about Selenium is trying to guess how much you want to wait for things to load and appear before making assertions and continue on with the test. For example, when you click the 'Edit Labels' link on a Confluence page, the edit labels input div becomes visible. So how long do I need to wait in the test for it to become visible before actually testing? The wait time you specify to the Selenium client is a maximum wait time, however you still don't want to be waiting too long for a genuine failure. The problem is when you run the tests locally, you usually don't have to make the wait for very long (1~2 seconds). However, this is a different story on Bamboo. We decided to bump the default wait to 5 seconds and for extra slow pages (like the edit page) to 10 seconds.
The confluence.editor.ajax.disable Property
There was one other problem that I encountered, but is purely specific to Confluence. There was this one particular test that continuously failed when running the tests in maven but not in Idea. WHY? The only difference was that Confluence was started up by cargo through maven.... It took me a very long and frustrating few hours to figure out why...
Up until now, we haven't been able to test the ajax requests (known as 'heartbeats') that Confluence sends for concurrent edits to the same page. During our other test builds, apparently this continuous heartbeat being sent during tests were causing problems. So they decided to add a nice system property to disable this for our tests and was included in the parent pom of our test runner module. And guess what my failing tests was actually testing? It was testing for concurrent edits on the same page and thus relied on the heartbeats being sent... I have learnt my lesson and will always make sure to check the parent pom before using it blindly!
5. Watch the Build go Green
Despite all the problems we had, the 9 tests we wrote are passing right now :) The next step would probably be to setup a Selenium build with IE.... which should also be quite interesting...
