End-to-End Selenium tests in Java

Introduction

Today we are living a continuous driven business environment. Your features have to be ready as soon as possible, especially in your production system. That's the way you make your customers happy. One big requirement is to ensure your application works.

That's the point when end-to-end tests come into play. E2E-Tests make sure that defined requirements work properly in your product. If you commit and build they will ensure that your customers can do what they requested.

Benefits

  • You know that your customers will not be frustrated about features not working
  • From the developer's perspective you know when you have broken something, avoiding a lot of hassle
  • When your build is green, you can be confident about the quality of your software

Implementation

I will present how to setup E2E-Tests and analyse different possible solutions in detail. I will also explain important details like how to speed up your tests.

Requirements

There are different frameworks and you need choose the right one for your company. Most of them at least do the same stuff: testing your application by simulating user actions by executing code. In my example I will show you how to use Selenium Grid. It's a e2e-testing framework for testing websites. It's open source and has big contributors like Google, Adobe and Mozilla. It's no assurance, but experience has told us that popular software like this will be stable, fast and flexible.

Selenium Grid

Selenium grid is a distributed, cross-OS executable Java program that contains different components. It contains a hub that controls the whole direction process.

There are also selenium nodes instances that contact the hub with their available capacity. The capacity is defined by available browsers and a possible number of simultaneous instances.

Your test contacts the hub to execute commands on a given browser. The hub will check if there is a browser available with the requested attributes. Test communicates to the hub like this:

"Hub, I need a Chrome version 43 running on Windows 7"

The hub will then look for an available scenario. If there are scenarios registered and ready then the tests will be executed on the matching node.

So it's possible to execute your tests on many machines simultaneously for different builds.

Set up a hub and a node

Every browser needs a driver. In windows the Firefox driver is still available in the Selenium jar. So you have to add a Chrome driver if you want to execute your tests in Google Chrome. In my example I will use Firefox and Chrome to demonstrate how to setup your environment.

First of all start your hub on a server or on your local machine:

java -jar selenium-server-standalone-2.44.0.jar -role hub

and start a node, adding the path to your chrome driver

java -jar selenium-server-standalone-2.44.0.jar -role node -hub http://localhost:4444/grid/register -Dwebdriver.chrome.driver=".\chromedriver.exe"

If it works your hub's output will tell you that an node has registered.

21:12:48.521 INFO - Selenium Grid hub is up and running  
21:13:16.785 INFO - Registered a node http://192.168.1.22:5555  

Run a test

The next step is to create a simple Selenium test. I suggest to use testng or junit to run your tests. In my example I'll setup a simple maven project and use testng to run my test. I'll also set up a test suite to execute them simultaneously.

You require the following libs in your pom.xml:

  • testng
  • selenmium-java

and you can use the maven-compiler-plugin and surefire-plugin to compile and run your test on a build.

Afterwards you are ready to go and can start coding like this:

    public void exampleTest() throws Exception {
        driver.get("http://localhost:9000");
        assertEquals(driver.getTitle(), "Welcome");
    }

Selenium pitfalls

I give some advice and best practice that you will need as soon as you start using selenium seriously.

Timeouts and waits

You should really avoid using any kind of static timeout. You have to figure out the potential triggers for your application. A common example is to wait until the page is completely loaded. In this case you can use some kind of JavaScript event. For example if you use jQuery you can just make use of your ready function like this:

((JavascriptExecutor) driver).executeAsyncScript(
                "var callback = arguments[arguments.length - 1];" +
                "$( document ).ready(function() {" +
                     "callback();" +
                "});"
);");

In this example you use an async script executor. Once your callback function does get executed your java code will go on. Note that this is plain blocking java.

Another common example is when something is loading. Imagine their is a div with a class name "splash" visible when something is loading. This time you can use Selenium's wait condition:

public void waitUntilLoadingDivDisappears() {  
     wait.until(
        ExpectedConditions.not(
           ExpectedConditions.visibilityOfAllElementsLocatedBy(By.className("splash")))
     );
}

This will block your java method to execute the next line until your condition is fulfilled.

Using Selenium's wait condition and async javascript executor are simple ways to create a good test without timeouts. Just make use of good events and profit from using your app's specific notification to create fast executing tests.

Parallel testing

By using for instance testng's suite definition, you can execute tests simultaneously. When you define a suite you have to enable parallel testing and define your packages or classes.

<?xml version="1.0" encoding="UTF-8"?>  
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">  
<suite thread-count="2" name="Suite" parallel="tests">  
    <test name="Firefox Tests">
        <parameter name="browser" value="firefox"/>
        <packages>
            <package name="e2e.*" />
        </packages>
    </test>

    <test name="Chrome Tests">
        <parameter name="browser" value="chrome"/>
        <packages>
            <package name="e2e.*" />
        </packages>
    </test>
</suite>  

Now your tests are executed twice. In managing different browsers and operating systems, the parameter system of testng is great. You can use the defined parameter browser to request a specific browser on a hub.

    @BeforeClass
    @Parameters({"browser"})
    public void setupBrowserAndTimeouts(String browser) throws Exception {
        DesiredCapabilities capabilities;

        if (browser.equals("firefox")) {
            capabilities = DesiredCapabilities.firefox();
        } else {
            capabilities = DesiredCapabilities.chrome();
        }

        driver = new RemoteWebDriver(
                new URL("http://localhost:4444/wd/hub"), capabilities);
    }

This will result in a parallel execution of your tests without limiting of concurrent executing tests.

Set boundaries for your scripts and waits

You will run into errors caused by side effects. To make sure that you tests do not block your node you can set a max timeout for executing scripts and waits.

    @BeforeClass
    public void setBoundaries(String browser) throws Exception {
        wait = new WebDriverWait(driver, 10);
        driver.manage().timeouts().setScriptTimeout(10, TimeUnit.SECONDS);
    }

This will cause your waits and script executions to limit their wait time by 10 seconds.

Ordered execution

By default, testng executes tests in a disordered way. There are different solutions that will result in an ordered execution. Compared to unit tests in e2e-tests, an ordered execution is useful. End-to-end tests are heavy in terms of execution time, so if you can save time by benefiting from an already existing state then you should make use of it. I have read about a generic method that performs over all methods, adding and incrementing priorities to force an ordered execution. A less strict way would be to use the priority or dependsOnMethod annotation property if required.

Refactor your e2e tests

The code examples look a bit technical. It's very important that you refactor your tests over time. You have to create a good structure to avoid calling waits and writing big queries. The result should be a readable and maintainable layer. Create a layer that helps you to improve over time and do not execute plain selenium methods. Not all developers will write e2e tests: primarily your QA will write them and they probably prefer writing in a domain-specific language. You don't need to create a custom language. Java has a good syntax and is nicely readable when well structured.

    public void calculateFee() throws Exception {
         search.field("category").set("A");
         search.start();
         grid.select(0);
         assertEquals(detail.field("fee").value(), 30);
    }

Behind the scenes it looks for an input field with the name category in your search area and sets the value to 'a'. Afterwards it starts the search and the first entry of your result grid is selected. It will be at least tested if the expected fee is correct.

Compared to the examples above this is a lot more readable. In practice it's not that hard to get this level of abstracting. Additionally you can optimize your layer over time without changing the api.

Test boundaries

In theory there are definitions that tell you what end to end tests, unit tests and integration tests are. Google tells us that good practice is to have a solid unit test base, some integration tests and a few e2e tests. In a concrete case it means you should test your app's happy path. You will profit a lot from knowing that your customer can do what they requested with your application. But don't test every special case of your app and never use it for testing and fixing defects. This will lead to a fragile and slow environment over time. The following rules help to identify if it's a requirement that should be tested:

Is it unlikely that it will fail after a patch made your test pass?

If your answer is 'yes', you should think about creating a special case test. By concentrating on testing the most used paths that fulfil the customer's requirements, you will have a very good cost-benefit ratio. Also you should avoid becoming slow and fragile. Maintainability and extendibility are very important and should be prioritized.

Are there other applications or parts of an application that have the same functionality?

If yes, you want to create an end-to-end test for a unit or an integration testable feature. Base functionality should be tested in a very specific context.

Summary

All in all, you will profit a lot by using end-to-end tests. You can maximize your benefit by customizing your testing strategy and infrastructure to fit the needs of your customers. You don't need to setup many Selenium nodes with different operation systems and browsers. Concentrating on gaining a lot from your time is the key to success. If you have successfully integrated your e2e-tests in your build process and you ensure that your core requirements work, then you are further ahead than the majority of companies. It's one important piece in the continuous delivery puzzle, bringing you increased software quality and ultimately greater trust.

Notes

Journerist

Read more posts by this author.