One reason that automated UI tests can be unreliable is that they tend to be sensitive to what else is on screen at the time and even things like the current screen size. Developers running the tests locally also find it annoying to have windows opening and closing on their machine while the test runs and are unable to do anything else because their clicking might interfere with the test. At LMAX Exchange we solve that by isolating tests in their own X session, created using vncserver. We simply start vncserver with:
vncserver :20 -geometry 1600x1200
Then set DISPLAY=:20 as an environment variable when starting WebDriver’s Firefox instance:
FirefoxBinary firefoxBinary = new FirefoxBinary();
firefoxBinary.setEnvironmentProperty("DISPLAY", ":20");
The Firefox window then pops up in it’s own isolated X session. We can
still use a vnc client to watch as the test runs but we can also let it
run in the background and continue using the machine for other things.
In CI it allows us to run UI tests on a headless server. Since we run a
number of tests in parallel, in CI we start a number of vncserver
instances and allocate a different one to each running test to ensure
they’re completely isolated. Simple, but incredibly effective.