Selenium - Is it worth the pain?
August 2, 2007 9:13 PMThought I might share with you a few of my experiences with using Selenium. It is a little overdue as I have only recently been able to bring myself to talk about it.
With JIRA 3.10, we introduced the AJAX-based 'User-picker' and 'Issue-picker'. A very handy feature, if I don't say so myself.
Having recently completed some AJAX eye candy for Fedex V, I found myself tasked to do this, along with one of our new recruits - Brad (a Javascript guru*). Our architecture is based on dwr for the communication and yui for all the Javascript goodness. We went a bit nuts, making the yui autocomplete do things it was never meant to do - subsections, titles, non-selectable items... We also implemented our own dwr datasource for yui that enabled yui and dwr to integrate seemlessly. More on that in my next post.
As a good friend suggested, Selenium seemed like the perfect tool for the job of testing this dynamic feature. We scheduled 16 hours for writing the tests - a couple of hours to write the tests, the rest for figuring out how to use it. This turned out to be a bit of an under-estimation.
We hit a few road blocks:
Key Events
I guess the the first sign of things to come was when we were having trouble testing the autocomplete pop-up i.e. we couldn't get it to pop. It turns out selenium.type(...) doesn't actually simulate a user typing into an input box. Go figure. So we typed in the text and triggered everykey stroke seperately - selenium.keyDown(...); selenium.keyPress(...); selenium.keyUp(...);. It worked!! in Firefox... In other browsers, it would now would print every character twice. So a common interaction such as typing some text into an input box turned into:
public void typeWithFullKeyEvents(String locator, String string, boolean reset)
{
if (reset)
{
selenium.type(locator, "");
}
char[] chars = string.toCharArray();
// Some browsers clear the text field when we start typing,
// so we need to pre-populate this string with the existing contents
StringBuffer sb = new StringBuffer(selenium.getValue(locator));
for (int i = 0; i < chars.length; i++)
{
char aChar = chars[i];
String key = Character.toString(aChar);
sb.append(aChar);selenium.keyDown(locator, key);
// some browser dont actually input any characters on these events
// supposedly to prevent JS spoof attacks. So we type for them
if (!SeleniumProxyUtil.getInstance().getUserAgent().equals("firefox"))
{
selenium.type(locator, sb.toString());
}
selenium.keyPress(locator, key);
selenium.keyUp(locator, key);
}
}
Other gotchyas - the key events will not give the field focus.
Timings
Due to our non-selectable items (section labels etc.), we wanted to test that some elements attributes do not change. We fire a mouse event on the element, and then use an xpath locator to check that the element's attribute did not change. Worked fine on my local box, though it needed a slight pause between the event fire and test on the Bamboo box. This was a very common fix for a lot of the things that would break - "Add/increase the pause".
xpath
So we used xpath locators for our tests. Seemed resonable. Right? Wrong! These xpath expressions are evaluated on the browser. Each browser constructs its DOM differently and may evaluate the expressions differently. We ran into trouble locating attributes on DIV items inside list items... The solution - write javascript locators. We ended up using yui to locate our things for us.
WTF?
To make things a little nicer for our users we decided to put in an autoscroll feature. I.e. when the pop-up pops, scroll the browser window to display the whole pop-up if it was hanging below the bottom of the screen.
Our tests were still a little flakey on Bamboo, though we had just had our greatest number of consecutive successful builds (5), so I wasn't overly surprised when they broke again. Only 3 out of 40 odd tests. After putting in/increasing numerous pauses, I still couldn't get them to pass. Why would scrolling the screen cause the tests to fail? We fire events on selected elements - not co-ordinates. The build kept on breaking...
We reached our lowest successful build percentage...
I decided to have a look at what was happening on the server (admittedly I should have done this earlier). To do this I needed to open an ssh tunnel to connect the vncviewer client to the vncserver. You can open a tunnel with ssh using this: {{ ssh -L 5901:localhost:5901 username@remote.host.com }} and then for vncserver instance 1, localhost will pipe through to remote.host.com. Increment the remote (second) port number by one to get to each successive vncserver instance, e.g. 5902 for instance 2. Leave the local (first) port number as 5901 so when you run vncserver as if to connect to your local instance 1 it will actually pipe through to remote instance 2.
So what did I see? I saw the mouse positioned directly in the midle of the screen, when the browser window scrolled, the mouse was positioned over one of the non-selectable items. Only 3 of tests caused the screen to scroll.
How to fix? We installed xwarppointer, to enable us to move the mouse pointer through bash, and tucked it away in a little corner.
Ahhh, the green!
Lessons Learnt
1) As much as you would like it to be, the Selenium client is not a user! It won't trigger all the events a user will and different browsers rely on different events!
2) Cross browser my arse!
3) Pauses are a Selenium Test's friend!
4) When running tests, keep the mouse the f&#k away!
Or, are we just using it wrong?
Was it worth it?
Tough to say. We are getting more used to all of Selenium's little quirks, though we are still struggling to get it to pass on IE and Safari. It did help us pick up a ThreadLocal leak that could have been quite bad, but is hasn't really picked up any UI bugs.
I guess I will put off a positive or negative conclusion until we build up our suite of tests, delve further into AJAXy features and have gotten used to Selenium some more.
Cheers,
Nick
JIRA Developer
* In the JIRA team anyone who knows what a namespace in javascript is considered a Javascript guru.
Copyright © 2009 Atlassian Pty Ltd.
17 Comment(s)
Hey Nick!
I had the same fun when trying to write a Selenium test that interacted with a file save input box. I had to use a java.awt.Robot to do the key presses to actually enter a filename in the input box as I think it is disabled as a security feature.
By Keith at August 3, 2007 1:07 AM
Nick - I've also experienced similar issues when working with Selenium and ajax; I usually end up using pause as well - but on occasion I have used the WaitForCondition method, of course this requires writing custom javascript and can be difficult to debug.
instead of using xpath for locators could you have modified your mark up to include ids on each control? maybe this was not resonable?
Keith - selenium is using javascript to manipulate controls on the page and there is definitly a security feature that does not allow this by default - a good thing as I would not want sites that I went to uploading files off of my hard drive.
The 'work around' I used was specifying '*chrome' for the browser string when constructing a DefaultSelenium object; this launches a version of FireFox with these security setting disabled
By House9 at August 4, 2007 4:17 PM
Hi, here are a few experiences that I had. Hopefully they will be useful for you
1) There are certain things that you just cannot test. I normally would avoid things like keyevent (maybe just me) and just call the function right away. Selenium allows you passing in Javascript code to evaluate. You do need to refactoring the Javascript function to make this possible.
2) Again, this might be just me. I have given up on xpath or things like that, and just use id most of the time. It might sound bad but I just don't have time to spend on figuring them out.
3) Pause is actually bad. Before you know it, you will have pause all over the place to make things much slower, not to mention your tests could become non-deterministic which is the biggest headache of all. There is a "waitForCondition" (or something like that) in Selenium, where you can pass in a Java script expression to evaluate. Selenium will keep evaluating it and return when that expression becomes true, or time out.
4) If you can avoid using events during the test, I think (operative word here) you can regain your mouse.
By Shane at August 5, 2007 4:36 AM
Do you have some sample code about your Timing problem ? Using selenium for a few month building a web application test framework (that works pretty fine), i don't understand your point. I've always been able to use the waitFor...() functions for not having to insert a pause in any of our 6000+ lines of test code.
By sebastien at August 7, 2007 6:52 AM
Well, you guys could dust off FatCow (http://opensource.atlassian.com/fatcow/index.html) and give Selenium a run for its money. ;-)
I suspect, though, you'll run into similar problems testing the UI with other tools. UI testing, in general, is a notorious tarpit. And testing Web UIs -- even more so. That's why there are a million testing tools -- no tool has ever gotten testing the UI 'right'-- at least not yet. With that said, sorry for your struggles with Selenium. The next version of Selenium (Core 0.8.3) does have a "typeKeys" command out of the box that will fire the proper JavaScript events in a field. My team at Google and I are also working on more seamlessly integrating java's Robot library where needed to deal with popups and at least make Robot available to test writers when the JavaScript-flavored type/typeKeys style of event emulation doesn't work. If you're using Java and Selenium RC you already start to use Robot now. The mouse issue you had-- I would consider that a bug in Selenium and should be fixed. Regarding the cross browser issues... Please file bugs... and preferably patches! :-)
Cheers,
Jason Huggins,
Selenium Developer
By Jason Huggins at August 7, 2007 9:50 AM
At the risk of sounding like an idiot, how did you get Selenium tests into Bamboo? Are you encapsulating them through JUnit or am I missing something?
-Matthew Janulewicz
By Matthew Janulewicz at August 8, 2007 6:06 PM
Jason,
I believe typeKeys command is available in S-Core v0.8.2 -- I do use it successfully. I just posted regarding a keyPress issue here:
http://blogs.telerik.com/blogs/konstantin_petkov/archive/2007/08/13/3129.aspx
It can help someone.
Nick,
I'm sorry to find such bad impressions of Selenium. I think you'll gradually like it.
Regards,
Konstantin
By Konstantin Petkov at August 13, 2007 6:16 AM
Matthew,
We use Cargo for our builds. We test all our products on multiple application servers.
By Scott Farquhar at August 14, 2007 5:35 PM
Hi guys,
Thanks for reading/commenting. Really should have responded earlier than this.
Just re-read the blog and I do bash it fair bit and the title was typical blog bait. I think I just got into groove and went with it...
Awesome news on the key type being addressed.
In regrads to the timing issues... For the vast majority we use waitForConditions but in some of the tests the waits were for things like testing for the absence of something.
E.g. when mouse hovers over something, you want an element to remove a class attribute. Due to our dwr/yui marriage (which I can't decide is a nasty or elegant hack) some things get set then removed. I.e. the class attribute would get set then milli-seconds later would be removed. Depending on the cpu load, our selenium test would test it during different stages and return differing results. (I am writing this of the top of my head and it was months ago, but think it was an issue like that).
We are getting used to it and writing tests is becoming easier as we build up our own libraries of helper classes and the ide in Firefox makes them fast to write. We still do run into problems and I still don't think we have it running under all browsers but it has been a while since I felt I was fight Selenium.
On saying that, we also still run into the odd problem using HttpUnit...
As we found with FatCow (the less said about that the better), writing testing frameworks is hard and fraught with danger. Selenium is one of the most ambitious of the lot and definitely has potential. Kudos go out for even attempting it. We do believe there is a need for a framework that can test AJAXy apps and at the moment we are putting our bets on Selenium.
I just really hope that the pains we did suffer were early adopter pains and that I am proven wrong in my rants.
By Nick Menere at August 16, 2007 12:11 AM
Nick,
I realize I'm late for this thread, but I'm struggling with similar problems. You wrote you used javascript locators that link into YUI to locate your objects. I'm trying to do the same thing with another JS library (qooxdoo). Are you at liberty to disclose a small sample how you achieved it?
Thanks, Thomas
By Thomas at November 14, 2007 8:46 AM
Dears,
I have a system and there are several parts where I have ajax autocomplete code. The system requires the user click in the popup menu, since clicking in the popup introduces the id of the selected object in the page. I can't capture this using selenium and I do not know how to solve this problem. Does Anyone have a suggestion for my problem?
TIA,
Pedro.
By Pedro at February 7, 2008 12:43 PM
@Pedro - did you ever resolve you issue? I'm working on a jQuery autocomplete field and having the same issues with Selenium.
By Jim Priest at February 27, 2008 11:33 AM
Ummm... "The next version of Selenium (Core 0.8.3) does have a "typeKeys" command out of the box that will fire the proper JavaScript events in a field. " But it doesn't work everywhere, does it? It works in Firefox, but not in IE. And nor does keyDown/keyUp...
By Tracey Annison at June 18, 2008 12:48 AM
Working with Selenium and YUI is a real pain, particularly if you're interested in testing on IE. IE has terrible xpath support, and you can't ensure that there are sane and meaningful ids on elements when you're using YUI. When all was said and done, we decided to drop IE testing and stick with Firefox.
I've found the easiest way to deal with YUI testing is to use CSS selectors. YMMV.
By Anonymous at August 13, 2008 6:57 AM
Hi
I am using selenium for Ajax application.My requirement is onChanging of text field i am calling a javScript function.my code is
When i record it selenium is not able to record calling of method.Because after
entering text is name textbox if i click anywhere on the page it will call that method. but Selenium is not recording it.
But when i call same method on click of a link then selenium i sable to record that method calling. Now my code is
click Here for calling My Method
What should i do so that selenium can record calling method in first case also.
By saurabh at October 9, 2008 11:53 PM
Nick, can you give us an update on where you guys are with Selenium these days? Maybe even another blog?
By Bob Swift at December 18, 2008 12:17 PM
A late Comment regarding autocomplete and or clicking pop up for future users, (Pedro's issue)
If your site requires a user to click an autocomplete box an easy way around this is a KeyDown/KeyUp on something like the Down arrow followed by keyDown/keyUp on enter or tab or program specific key.
If you give up on Selenium and it's timing issues check out WebDriver, although they are integrating with selenium soon for Selenium 2
By Valchris at June 2, 2009 6:04 AM