Guides

Setting up crossbrowser testing of Javascript with TestSwarm and Browserstack

April 9, 2013

Okay, this one is not free as in free beer; Browserstack requires a paid subscription.

Let’s assume we have a test suite for Javascript written, say, in Jasmine. It resides in our web app under the humble /site/tests path. And it’s a good idea to have this test suite run:

  1. cross-browser;
  2. periodically;
  3. and publishing reports for each test run on each platform.

jQuery team has built a tool for it all, called TestSwarm. It’s a server which organizes runs of test suites and stores results of such a runs. However, TestSwarm has a catch: it does not run anything itself, being only a kind of storage area. For test suite for some desired browser to be run, this browser should connect to TestSwarm, get the test suite offered and run it itself. So, we have a problem of where to get the browsers.

For the supply of browsers we’ll just use the Browserstack. They have an API for most things and there exists a bridging Node.js module called testswarm-browserstack which does exactly what it’s name suggests. Yesterday I just have completed a Jasmine/TestSwarm/Browserstack combo on one of our projects and I think I’ll write about how it was done.

I assume all libraries used here will be horrendously changed in future, so I warn you that instructions are relevant as of 2013.04.09.

End result

Each night, TestSwarm instance will schedule a test run of your test suite on an array of browsers. Browsers required will be requested from Browserstack. They will run your test suite in their respective environments and TestSwarm will store the results gotten from this test runs as a plain HTML pages for you to review later.

Observant hardcore CI-users probably noticed that there is no post-commit hooks. Indeed; for per-commit test runs we need to be able to make a fully automated fresh install of our web application and so TestSwarm can point browsers to the test suite on the web app instance raised just for this tests. It will add a couple of additional steps to this scheme.

Summary

  1. Make your test suite publicly available on the website.
  2. Insert inject.js to your test suite.
  3. Install the TestSwarm instance somewhere and configure it.
  4. Install and configure testswarm Node.js module.
  5. Install and configure testswarm-browserstack Node.js module.
  6. Fiddle with test.js script from testswarm module and schedule it to run nightly.
  7. Fiddle with run.sh script from testswarm-browserstack module and schedule it to run constantly.

Getting your test suite prepared

Obviously you will need a test suite, as I already mentioned in very second paragraph. In my case it was a Jasmine-based test suite indeed, placed under /site/tests. Exact placement is not important, URL just needs to be publicly available.

Now you need to modify your test runner HTML page to insert a call to special inject script from TestSwarm.

It’s explained in the relevant documentation for inject.js but I repeat it here anyway.

This file detects which unit test framework is present on the page and registers a callback for when that framework indicates it is done. The script then extracts the test results and submits them to TestSwarm, using the identification given from the TestSwarm context (run-id, client-id etc.).

You may unconditionally inject.js, it will only act if it detects that it is being run in a TestSwarm environment (so you don’t need a seperate version of the test suite “for TestSwarm”).

Most important part: the inject.js file should be included in the head element after the unit test framework of choice (as it needs to register a hook), but before the test suite starts.

At the period of writing this article issue #167 regarding Jasmine support in TestSwarm is still not closed so I had to hack a proposed solution into the script by hand. Anyway, I think it’s a better choice to copy the inject.js from the TestSwarm to your codebase. This way you’ll reference it not from remote domain, but from your local assets. It can be important if you assemble/compress your scripts or have an AMD system like Require.js.

After you place the inject.js (capable of detecting your test framework) into your test suite runner after the framework but before test cases, you’re done.

Installing TestSwarm instance

This step is probably the most straightforward of all. Just follow the installation instructions from the official repository. Do all the steps listed there, including the setting up the crontab for running cleanup action, all of it’s pretty important.

Regarding the localSettings.json configuration I’ll add from myself that you always can consult the caniuse.com to determine which browsers to add to your test array.

After you set up the TestSwarm instance, you’ll need to sign up to it, and remember three important pieces of information:

  1. URL of your user at swarm. It should be in form of http://SWARM_DOMAIN/user/USERNAME.
  2. Your created username, obviously.
  3. Your auth token (!) which you should pluck manually from the TestSwarm database (!!) after you sign up. It’s stored in your user record in the auth field.

Scheduling the jobs in TestSwarm instance.

By the way, this and the following steps can be performed on yet another domain, different from the one holding TestSwarm instance.

First you need to install a special testswarm Node.js module. It’s being done just by issuing npm install testswarm. I assume you do have Node.js installed already.

Now we’ll construct a job-adding script.

As I already said in the End result section, jobs will be added nightly and the idea is to just call a job-adding script from crontab. testswarm module has a sample script called sample-test.js which contains all we need but probably requires some fiddling from your side.

I ended up with the following script:

var testswarm = require( "./lib/testswarm" ),
runs = {
    main: "http://MY_APP_UNDER_TEST_DOMAIN/site/tests"
};

testswarm.createClient( {
    url: "http://MY_SWARM_DOMAIN/"
} )
.addReporter( testswarm.reporters.cli )
.auth( {
    id: "MY_USER_NAME",
    token: "MY_USER_AUTH_TOKEN"
} )
.addjob(
    {
        name: "Regular check @ " + (new Date).toUTCString(),
        runs: runs,
        browserSets: ["most-current"],
    },
    function ( err, passed ) {
        if ( err ) {
            throw err;
        }
        process.exit( passed ? 0 : 1 );
    }
);

Note that I asked to add job with browserSets containing “most-current”. Browser set with this exact name should be defined in your localSettings.json configuration for TestSwarm instance!

Basically, if you’ll need a post-commit hook, you will need to somehow auto-generate this script or pass some additional commandline arguments to configure the following things:

  1. MY_APP_UNDER_TEST_DOMAIN — most possibly you’ll need to create a separate domain with fresh install of your app or differentiate instances in some other way;
  2. Job name — most possibly you’ll want to name your jobs depending on commit they were scheduled from.

Scheduling the supply of browsers to TestSwarm

And now, to the final part of the setup. What’s left is to setup a script which will continuously poll the TestSwarm instance and if there’s a jobs in need of browsers ask them from Browserstack.

First, install the testswarm-browserstack Node.js module by issuing npm install testswarm-browserstack.

After that, you need to configure the settings by copying the sample-config.json to config.json and putting the Browserstack credentials and TestSwarm instance URLs to there.

Please note that in the testswarm.runUrl config paremeter your user endpoint is located. This means that if you have a several projects and want to differentiate them by usernames (as it’s seemingly intended by the TestSwarm authors), you will probably need to setup a separate testswarm-browserstack module instance for each project.

With configuration in place, you need to copy sample-run.sh to just run.sh and change the paths in it according to your installation of testswarm-browserstack. This run.sh script you need to launch periodically with probably 2-5 minutes interval. There is two reasons for such a constant polling even if you have only one scheduled TestSwarm job per day.

  1. You definitely can’t be possibly satisfied with just nightly builds and will change to per-commit runs someday.
  2. You can have jobs scheduled manually. All that’s needed for it is to launch by hand the test.js script from testswarm Node.js module we configured in previous step. For manual runs you probably don’t want to bother with running testswarm-browserstack’s runner manually, too.

For any details consult with the install documentation at the testswarm-browserstack repo.

Please note that it’s probably very important that no two run.sh scripts will be running simultaneously. It’s noted both in the documentation and in the sample for this script.

And that’s it!

Congrats.

Previous: How to set EMACS fonts rendering the same as the other KDE applications Next: Real-world Acceptance Testing with Behat