Reality-Loop

Top 24 Agile Web Testing Tools

October 07, 2010 | 13 Minute Read
This post was originally posted on my old blog.

hand-tools-list-important.jpg According to the book Agile Testing there is a certain kind of tools called Agile Web Testing Tools. The book does not contain a precise definition.
My definition of Agile Web Testing Tools contains the following characteristics:
  • Lightweight (not needing heavy infrastructure or installation nor difficult licences)
  • Can be run on any developer workstation and can be easily integrated in a continuous build
  • Programmatic API for web automation
  • Integrates with common test automtation frameworks (JUnit, NUnit ...)
I don't fully agree with the "Agile" prefix, but I guess that is an attempt to differentiate those tools from heavier multi purpose tools like Borland SilkTest, HP QuickTest (formerly Mercury) or Visual Studio Team System Test Edition.

Following from the characteristic of hose tools is the fact, that they are rather developer focused. They approach functional web testing from the unit-testing perspective and strive to be as close to the existing programming artifacts (the tools have a programming API, tests are written as code, tests should be runnable as part of the normal build, no special management tools are necessary ...)

The current situation around this type of tools is quite entangled, some tools are extending or complementing others, some are competitors and new ones are sprouting out of others...

The following post tries to get some order and structure in the landscape of those Agile Web Testing Tools.

The first imortant distinction is between out-of-browser (headless browser) and browser automation tools.
  • Out-of-browser tools do not run the test against a real browser but use some kind of browser-simulation/emulation. This simulation/emulation can usually be run in-process, embedded inside an automated test.
  • Browser automation tools drive a real browser to access the application under test.

The browser automation tools then can be divided between browser-remoting tools and in-browser tools.
  • A browser remoting test-driver controls the browser from the outside through some programmatic API.
  • An in-browser test-driver is a test execution runtime that runs inside the browser (usually as JavaScript). This runtime then accesses the application under test from within the browser.

Some confusion is currently created by a third kind of tools. These tools are wrappers or abstractions over concrete test-drivers. They try to give a common programming model for automated functional web tests. This way the same tests should be runnable on a headless browser or on a browser automation driver.


headless.png Headles browsers:

HtmlUnit (Java)
HtmlUnit is a headless browser that runs in in memory and is implemented in Java. HtmlUnit is the base for a lot of other libraries/tools.
HtmlUnit itself uses Apache HttpClient, a library for from Apache.
HtmlUnit includes Rhino as JavaScript runtime. Unfortunately Rhino is not used in any browser, so if you test JavaScript using HtmlUnit the results may differ significantly from those browsers.

There are attempts to run a IKVM-ported version of HtmlUnit under .NET:

HttpUnit (Java)
An alternative to HtmlUnit, but the last release is from 2008. This project seems to be dead.

Webrat (Ruby)
A browser simulation for testing ruby web applications. Primarily used for testing Rails, Merb or Sinatra applications.
Because Webrat is a browser simulation, it is faster than in-browser testing solutions. The drawback is, that it does not handle JavaScript at all.
Webrat also can be configured to run on top of Selenium. In Selenium mode the tests are run with selenium against a real browser, but the same webrat API can be used as in the browser simulation mode.
In Selenium mode Webrat takes care of running the java selenium server.
Webrat is wildely used in combination with the Ruby testing libraries Cucumber and RSpec.

Rack-Test (Ruby)
A library that mocks http-request for Rack applications. Rack-Test is usually used in combination with Ruby web framworks like Rails, Merb or Sinatra.

JSFUnit (Java)
JSFUnit is a test framework for JSF applications. It is designed to allow complete integration testing and unit testing of JSF applications using a simplified API.
JSFUnit tests run inside the container. They have to be packaged and deployed with your application.
JSFUnit is built on top of Cactus (a java in-container testing framework)



Aurora9.jpgBrowser Automation Tools

Watir(including FireWatir and SafariWatir) (Ruby)
Watir is a driver for IE, Firefox, and Safari. Watir runns on Windows, Mac and Linux. Obviously IE and Safari are only available on Windows respectively Mac.
The IE and Firefox drivers are stable.
Technically IE is driven thorough the COM-Interop. Firefox is driven through a Firefox-Plugin.
The Safari driver is experimental.

Watij - WebSpec (Java)
Watij was Initially a port of Watir to Java.
It seems to have become a project of its own. The latest release is called WebSpec and features a new and unique API.
Implemented in Java it runs on Windows, Mac and Linux and supports remoting of IE, Mozilla and Safari.
There is also a JRuby API for it.

WatiN (.NET)
WatiN is a port of Watir to .NET. Implemented in .NET it offers a browser automation API.
It runs on Windows and supports remoting of IE.
The newest release also supports remoting of Firefox through a Firefos plugin.
Latest release is from December 2009, which is a bit scary...

Selenium (Standalone, Integrating with Java, .NET, Ruby and more)
Selenium is a very sophisticated web-testing solution.
It is an in-browser test driver. It automates the browser to run the test steps. This automation is run from within the browser itself. The runtime for automating the browser is actually a JavaScript library that is loaded  into the browser. This runtime then automates the browser interactions which are executed in another frame or window.
On the client the Selenium-RC-Server (remote control) is responsible for controlling the in-browser runtime. The Selenium-RC-Server is is implemented in Java. So to run the server, you need Java installed.
Finally your tests written in any supported platform (Java, .NET, Ruby ...) are interacting with the Selenium-RC server.
Selenium supports automating all major Browsers (IE, Firefox, Chrome, Safari, Opera, Konqueror and more). Since the runtime is executed in the browser, JavaScript has to be enabled.
Selenium is very powerful. However it comes with a certain complexity and fragility since you depend on the correct interactions of several involved components.
Selenium also comes with SeleniumIDE, a Firefox Plugin to record interactions with the browser. Note: The interactions recorded in Firefox can also be run against other browsers later.
Another component is Selenium Grid, an infrastructure which enables to run Selenium tests on multiple servers against different browsers in parallel.

Canoo WebTest (Standalone, Java, Groovy)
Canoo WebTest is a free open source tool for automated testing of web applications in a very effective way.
WebTest is built on top of HtmlUnit. Tha means it is an out-of-browser testing tool.
Since it is not an in-browser testing-tool, you cannot be sure that the browser is behaving exactly the same, but HtmlUnit is very mature and has reliable JavaScript support.
Test steps can be written in xml or in Groovy. Those steps then can be easily combined to test scripts. Almost no programming skills are required.
Canoo Web Test integrates with Ant and can be run as standalone tool.
Very nice reports out of the box. If a test is failing you get a screenshot of the last page where the failure occured.
You need Java installed.
Here is a good comparison between Canoo Web Test and Selenium.

CodedUI (.NET)
Visual Studio 2010 comes with CodedUI.
With CodedUI you can create tests that automate the browser.
All the tutorials show how to use CodedUI with a Recorder and with Visual Studio TestManager.
But behind the scenes CodedUI seems to be a purely programmatic API: the recorder generates C# code (although not very readable).
But theoretically the programatic API of CodedUI can be used to write UI-automatin tests from scratch. However I think an abstraction layer on top of CodedUI would be needed to write those tests effectively.

Lightweight Test Automation Framework (.NET)
For an introduction see this blog-post by Steve Sanderson.
The Lightweight Test Automation Framework is a framework to test ASP.NET applications. It works with WebForms and MVC).
Tests are written in any .NET language, deployed alongside with your application and then run in the browser.
Since test execution is started through the web-interface, running the tests in a automated build is a bit tricky.


Windmill (Standalone, Python)
Windmill is a web testing tool designed to let you painlessly automate and debug your web application.



egg-wrap-su-642319-l.jpgWrappers / Abstractions:

Celerity(JRuby)
Celerity is a JRuby wrapper around HtmlUnit.

Culerity (Ruby):
Culerity integrate celerity and cucumber running on ruby
Culerity is a wrapper around Celerity to allow it to be called from MRI
You need to have JRuby installed.

Capybara
Capybara is a DSL for interacting with a webapplication. It is agnostic about the driver running your tests and currently comes bundled with rack-test, Culerity, Celerity and Selenium support built in.
Capybara provides a common API (the DSL) that is then run either on rack-test, Culerity, Celerity or Selenium. The newest addition is a driver that runs on env.js, a JavaScript environment, that runs out of the browser.
It seems to become the de facto choice in combination with Cucumber, replacing Webrat.

WebDriver / Selenium2 (Java)
For an explanation what the relationship between WebDriver and Selenium2 see the Selenium documentation.
WebDriver is another wrapper around other web-testing frameworks that provides a consistent API how to interact with web applications.
WebDriver has been merged with Selenium 2. The resulting situation is not very clear yet...
The project seems promising but is still very much under development.
WebDriver currently provides four different driver implementation: HtmlUnitDriver for out-of-browser testing, FireFoxDriver, InternetExplorerDriver and ChromeDriver for the respective browsers.
WebDriver is mainly a Java project, however a .NET version exists, however it is not very clear which features are supported in .NET.


JWebUnit (Java)
Wrapper around HtmlUnit and Selenium, providing a unified Java API. In this regard the same goal as WebDriver.

Watir-Webdriver
This is watir sitting on top of webdriver. This allows you to use the Watir-API and control any browser that is supported by WebDriver.

Tellurium (Groovy)
Tellurium is a UI module-based automated testing framework for web applications. The approach it takes is to describe the UI declaratively in modules and then formulate test steps against those modules. When the web-site changes you only have to update the modules, not the steps.

MVCContib (.NET)
The MVCContrib projects contains some abstractions over WatiN for testing ASP.NET MVC applications.
These are helpers in MvcContrib.TestHelper.Ui and WatinDriver.
However the documentation of those helpers is poor. The best place is to look at the CodeCampServer project and Chapter 20 in the book ASP.Net MVC 2 in Action.



bottles.jpg Other libraries (for homebrew web-testing):

Mechanize
The Mechanize library is used for automating interaction with websites.

Nokogiri
An HTML, XML, SAX, & Reader parser with the ability to search documents via XPath or CSS3 selectors.
Implemented in Ruby.
Usually you don't use Nokogiri directly in your automated web-tests. However this library is used by a lot of the higher level Ruby web-testing libraries mentioned here.

hpricot
Hpricot is a library with a Ruby-API for parsing HTML. It is written in C and therefore very swift.
Hpricot is an alternative to Nokogiri.
As with Nokogiri you don't usually use Hpricot directly in your automated web-tests. However this library is used by someof the higher level Ruby web-testing libraries mentioned here.


I would be interested in your experiences with web testing tools. Do you know other tools that fit the characteristics of  "Agile Web Testing Tools"?

http://www.google.com/s2/favicons?domain=twitter.com follow me on twitter, I need some friends :-)