Guides

Real-world Acceptance Testing with Behat

May 31, 2013

Okay, today let’s say you have a working webapp providing some functionality to it’s users. And, let’s say you test this functionality manually. You have the long document describing how things should work and on each iteration you sit down, open your application and start clicking and filling forms and checking that all is going according to specs.

Of course this is pure horror, but it’s something that should be done to be sure that after all changes in current iteration your features aren’t broken, so, you don’t have regression errors.

Of course, there is a clean and obvious solution to automate this process.

Such automated tests are commonly called “acceptance tests” and their purpose is to imitate real user who uses the application completely through it’s UI. They’re slow, but it’s better than to do it by hand. There is the tools to make this automation work.

Today I’ll speak about the acceptance testing by Selenium, supported by the power of Behat test harness.

Please note that the setup I’ll be describing here was implemented on *nix system and most possibly will be a pain to replicate on Windows or MacOS. Also, as the test harness used is a PHP-based Behat it’s assumed that you’re going to test a PHP-based application.

End result

In the end, you’ll get the following:

Acceptance tests which will run either on server or on your development machine either in visible mode or completely in background, exactly like the automated tests should behave.
This tests will be an imitation of a real user behavior on the website.

Prequisites

You should have an terminal access to your server, preferably root or at least be able to install things and manage users.
Your server should have enough capacity to run a single instance of full-blown Firefox or probably other desktop browser.
Your application should ultimately be Yii-based, because most of further explanations of configuration will be Yii-specific.
You should understand what Behat is and how to use it in general, as here is only the configuration and setting it up is explained, but the one writing the actual tests will be you.

Short summary for impatient

Make the test entry point to your app, which uses separate database.
Publish this test entry point under separate domain.
Download and put the Selenium jar file to your codebase.
Download and put the Behat, Mink and Mink Extension phar files to your codebase.
Setup the Behat test directory in your codebase, probably with behat --init command.
Write the proper Behat config which binds Behat, Mink, Mink Extension and Selenium together.
Write the test entry point domain to the Behat config.
Write some dummy test scenario just to check that everything is working.
Install Xvfb.

After that, you are able to run your Behat test suite completely off-screen by running:

$ Xvfb :1 -screen 0 1024x768x24
$ DISPLAY=:1 java -jar ./path/to/selenium.jar
$ php ./path/to/behat.phar --config=./path/to/behat.yml

NOTE: Given that you are under user which can launch desktop browser successfully, i. e., has a home directory!

Preparing the test version of application

First of all, you should understand that when you run acceptance tests, your tests and your web application are completely detached from each other. As a result, you can check the results of your actions only by side effects directly visible on the web page under test. Only thing which can really be shared is the database.

Second, you definitely don’t want your tests to fiddle on the real-world website, especialy if it’s already published in the WWW and people visit it regularly. Ideally, you should re-deploy the whole website to test domain before each acceptance test run, so tests can, for example, fill the database with controllable data. This is hardly achievable, of course.

That’s how we handled it in the Bandwaggon:

We made a clone of our production database without the real data in it. It’s a database which will be used by application under test.
Alongside with the normal db component configuration we added a configuration for second database connection named testdb, which is for this clone of production database.
Alongside with normal index.php entry point we created it’s copy, named index-test.php, which is mostly the same, just the config which is used to create the WebApplication instance is not the normal one, but test one.

Made a special test config, which has several specific tunings like setting all foreign interfaces to sandbox mode and most importantly, replaces the configuration of main db component with another for component named testdb.

Fourth step was done like this:

     <?php
     /**
      * Special configuration for when the application runs under acceptance tests
      */
     return CMap::mergeArray(
         call_user_func(function () {
             // Replace main db connection with testdb for tests.
             $main = require __DIR__ . '/main.php';
             $main['components']['db'] = $main['components']['testdb'];
             return $main;
         }),
         array(
             // ... other small tweaks for test mode ...
         )
     );

So, it’s just a direct brute replacement of one component’s config with another. But that’s not all.

This index-test.php entry point is made into a separate domain by making a Apache virtual host directive like the following (that’s the setup for my personal local workstation):

 <VirtualHost *:80>
   ServerName tests.bandwaggon.my
   DocumentRoot /home/hijarian/projects/bandwaggon/codebase/frontend/www
   ErrorLog /home/hijarian/projects/bandwaggon/logs/test_error_log
   CustomLog /home/hijarian/projects/bandwaggon/logs/test_access_log combined
   <Directory /home/hijarian/projects/bandwaggon/codebase/frontend/www>
     Options FollowSymLinks
     #
     ## Here's the important stuff
     AllowOverride None
     DirectoryIndex index-test.php
     RewriteEngine on
     RewriteCond %{REQUEST_FILENAME} !-f
     RewriteCond %{REQUEST_FILENAME} !-d
     RewriteRule . index-test.php
     ##
   </Directory>
 </VirtualHost>

The goal is to access the application by the URL http://your.test.domain/ and see that it’s totally like your real app but is running using the test database. Most obvious sympthom will be, of course, the total absence of user-generated content.

The reason behind the separate test domain is to hide the index-test.php entry point completely. Any user agent connecting to the test domain will interact with the test application just like with the real application, which is the whole point of end-to-end tests we are building here. It is especially important for AJAX calls.

If you will be completely re-deploying your application to the test domain, you’ll not need both the RewriteRule/DirectoryIndex fiddling and the separate “test” configuration, because with such a setup you configure environment automatically on each deploy anyway.

Selenium setup

This is really simple step. You will be using the modern Selenium Server distributed as .jar file, so just download it from the official website and put into your codebase somewhere. Traditionally the filename should be selenium-server-standalone-#.##.#.jar and the size of ~30MB.

You should probably check whether your system is able to run this jar by launching it:

java -jar selenium-server-standalone-X.YY.Z.jar

After you launch Selenium, it’ll be running as a daemon listening to requests. It has a very thorough API which will be used by Behat.

Behat setup

We here at Bandwaggon use the Behat in form of PHAR archives. You’ll need three of them.

First one is the main behat.phar, containing the essence of this tool. Second is the mink.phar, and it contains all the bindings to various tools which can be used in acceptance testing, with Selenium among them. Third and probably the most important is the mink_extension.phar, which is an addition to the bare Mink adding predefined test steps and more simple configuration of Selenium bindings.

All of them can be found either by skimming through the official Behat website or by directly accesing the downloads section.

In a small chance of misconfiguration between this three PHARs, you always can checkout the official repositories of Behat, Mink and MinkExtension and build the archives manually using the scripts provided there.

Put this three files to some directory under your codebase, too.

After that, you prepare the usual directory for Behat tests, by issuing the usual

php ./path/to/your/behat.phar --init

And after that you should create the behat.yml configuration file for Behat. For the setup described in this article configuration should look like this:

    # Global parameters, baseline for all profiles
    default:
      extensions:
        ./relative/path/from/this/file/to/mink_extension.phar:          # note the absence of quotes!
          mink_loader: './relative/path/from/this/file/to/mink.phar'    # note the quotes!
    # You absolutely must define the base_url parameter for this extension.
    # This will be done in local override below
          default_session: goutte
          goutte: ~
          javascript_session: selenium2
          selenium2: ~

    # Local overrides here
    imports:
      # You can add anything here, but you must define your local `base_url` value
      # for tests to run at all (see behat-local-example.yml)
      - '/behat-local.yml'

You should note several points here.

Both paths to mink_extension.phar and to mink.phar are relative to the directory where the Behat configuration is, just like with the paths in CSS files.
You use the full path to mink_extension.phar as a “name” of the Behat extension used.
You use selenium2 driver only for javascript sessions (scenarios marked with @javascript tag), this is so that you’ll still be able to write relatively fast-running acceptance tests when you don’t need the full-blown browser.
There is the parameter individual for each developer so it was moved to separate config file named behat-local.yml. Note that the path to it has a leading slash.

Your behat-local.yml should look like this (at minimum, of course):

    default:
      extensions:
        ./relative/path/from/this/file/to/mink_extension.phar:
          base_url: "http://your.test.domain/"

Note that we repeat the “name” of Mink extension here verbatim.

Testing the basic Behat/Selenium combo

Now let’s check whether we did the initial setup correctly. I assume that your workstation has a browser installed.

Writing dummy test

As your Behat installation contains the Mink Extension now, you can make your FeatureContext extend not the BehatContext but the MinkContext, this will give you quite large set of built-in test steps for using in your tests.

Write something absolutely basic as your first feature, like:

Feature System check

@javascript
Scenario Line "Home" should be somewhere on the homepage
  Given I am on "/"
  Then I should see "Home"

Running dummy test

Fire the Selenium in one console window:

java -jar ./path/to/selenium.jar

then fire up the Behat in second console window:

php ./path/to/behat.phar --config=./path/to/behat.yml

You should see how your main browser (Selenium tries Firefox by default) launches, opens your test domain and closes. In the Behat console should be the report about the tests either succeeded (in case you do have a string “Home” somewhere at landing page) or failed (otherwise).

Background run

You probably noticed that Selenium fires up the Firefox for real, taking focus away from you and in case you have a lot of test scenarios you basically cannot do anything while it’s working.

More than that, obviously it requires a monitor to work. On your workstation it’s not a problem but how about running the tests on the server, which does not and probably will never have a monitor attached?

It seems that X server contains one very useful tool exactly for such situations, called X Virtual Framebuffer, or Xvfb in short. It’s just like the X server but it does not try to really display anything on real monitor, making all processing purely in memory. X programs can be instructed to use Xvfb instead of normal X display and as a result will be running without the need in monitor.

This technique is an adapted version of the trick Jordan Sissel did with the Firefox. The trick is as follows:

Xvfb :1 -screen 0 1024x768x24
DISPLAY=:1 java -jar ./path/to/selenium.jar

After that Selenium will use the forged off-screen framebuffer as a monitor. This will allow us to launch firefox and work with it in headless system like web server. If you launch Behat after this trick it should not show anything on your screen and if being run on the web server will happily proceed to running tests.

Of course you need to install the Xvfb on your server, along with the firefox.

Specifics of running firefox on the headless web server

Your web server most probably will not have the normal users defined on it, just the root and the apache host user named either apache or www-data. Of course if you’re going to be really cool, you have something more intricate than a single Apache but it’s not important now. Point is, you’re going to fire up real desktop browser for your acceptance tests, so you need a real user in your system. The reason is that Firefox, for example, will try to create a lot of profile and configuration dotfiles in the home directory of user it’s running under but if it’s launched by the web server user most possibly it’ll not find anything like the home directory. And of course you don’t want to clutter your root home with volatile browser-specific stuff.

So, you ultimately need to add another user named like tester or so and run the tests under it. It was found by experimenting that you only need to run the Behat as this separate user, Xvfb and selenium can be launched from any user account. It seems that the one who launches browser is the Mink.

Setting up the cleanups

Well, at this point we already have working framework for launching Behat-based end-to-end acceptance test suite in background.

However, we lack one quite important thing, that is, shutting down Xvfb and selenium after the test run. You probably don’t want to either control manually whether they are running already before each test run or find & kill them utilizing the low-level power of ps -A | grep $FOO | awk '{ print $1 }' | xargs kill.

As we already decided that we’re on some *nix system, we can use some more console tricks. With this tricks we will store the PIDs of both Selenium and Xvfb in special temporary files and kill them after the test run restoring this PIDs from this temporary files.

It can be done like this:

Xvfb > ./path/to/Xvfb.log 2>&1 & echo $! > ./path/to/Xvfb.pid
java -jar ./path/to/selenium.jar > ./path/to/selenium.log 2>&1 & echo $! > ./path/to/selenium.pid

As a bonus, this command lines include logging of the output to files Xvfb.log and selenium.log, respectively.

Note the magic incantation & echo $1 > ./path/to/program.pid. It tells the shell to run whatever was preceding it in background and store it’s PID to file ./path/to/program.pid. $1 is a special Bash variable holding the PID of last executed process. You can see that I assume you’re running Bash here. If you’re the proponent of some other shell, you have to figure out the source of PID on your own.

After your test will be run, you shutdown both Xvfb and Selenium with the following über-magic:

kill `cat ./path/to/program.pid` && rm -f ./path/to/program.pid

So, as both .pid files contain just the IDs of programs, we can just feed them to kill directly. It’s not really important to erase the files which stored PIDs, but as they now hold the incoherent information, it’s better to purge them out of existence.

Complete console session with the test harness

Ultimately, your test run will proceed like the following:

$ Xvfb :1 -screen 0 1920x1080x24 > ./path/to/Xvfb.log 2>&1 & echo $! > ./path/to/Xvfb.pid
$ DISPLAY=:1 java -jar ./path/to/selenium.jar > ./path/to/selenium.log 2>&1 & echo $! > ./path/to/selenium.pid
$ sudo -u tester php ./path/to/behat.phar --config=./path/to/behat.yml
$ kill `cat ./path/to/selenium.pid` && rm -f ./path/to/selenium.pid
$ kill `cat ./path/to/Xvfb.pid` && rm -f ./path/to/Xvfb.pid

Of course, it’s better be saved as a configurable script:

#!/bin/bash
# Full Behat test harness run in off-screen mode

XVFB_LOGFILE=./path/to/Xvfb.log
XVFB_PIDFILE=./path/to/Xvfb.pid
SELENIUM_LOGFILE=./path/to/selenium.log
SELENIUM_PIDFILE=./path/to/selenium.pid
SELENIUM_JARFILE=./path/to/selenium.pid
BEHAT_PHARFILE=./path/to/behat.phar
BEHAT_CONFIGFILE=./path/to/behat.config
SCREEN_RESOLUTION=1920x1080x24

Xvfb :1 -screen 0 $SCREEN_RESOLUTION > $XVFB_LOGFILE 2>&1 & echo $! > $XVFB_PIDFILE
DISPLAY=:1 java -jar $SELENIUM_JARFILE > $SELENIUM_LOGFILE 2>&1 & echo $! > $SELENIUM_PIDFILE
php $BEHAT_PHARFILE --config=$BEHAT_CONFIGFILE
kill `cat $SELENIUM_PIDFILE` && rm -f $SELENIUM_PIDFILE
kill `cat $XVFB_PIDFILE` && rm -f $XVFB_PIDFILE

DO NOT FORGET that this script should be run from your tester user (note the differences in the line starting with php)!

Helper console command for Yii

To not bother with the shell scripts and have a consistent commandline interface the test session described above was wrapped in special Yii console command, which we called simply a BehatCommand. With this command full test suite run looks like this:

./yiic behat prepare
./yiic behat run
./yiic behat shutdown

The full source code of the BehatCommand are published here for anyone to use.

Congratulations

This article was based on the practical experience with setting up Behat/Mink/Selenium combo on a single particular project. Nevertheless recommendations was general enough so most possibly you will be able to adapt them to your specific needs.

Most project-specific part is how to setup the test double of your application. After you have the test domain which you can freely fiddle with then all remaining stuff is really straightforward to do.