Repent! Automated Web Testing is Inevitable

| Comments


Our job as web developers is to build defect-free sites. That job is getting harder. As sites become bloated more feature-rich, verifying everything in multiple browsers before it goes to QA becomes impossible. The only way to keep afloat is to allow automated web testing frameworks shoulder the burden.

Test EVERY Browser Your Site Supports

Yes, the line between tester and developer is blurring. The fact is that you really can’t call something “done” if you haven’t verified it yourself first. That means checking all the browsers your site claims to support, even the dinky ones.

“But my job is to build, not test!”, shouts the programmer. “My job is to verify, not to sniff out all browser compatibility issues you were too lazy to handle”, replies the tester.

It’s a balancing act, and as professionals we need to commit to delivering verified code. In the end, there are more feature and browser combinations than any developer could possibly check in a reasonable amount of time.

Sign In - 1990s Style

Back in the day, sites were still pretty simple, especially the trusty sign-in form.


About as simple as it gets. A couple of fields and a button and maybe a blink tag or two.

If we count features, that’s:

  1. Show the form
  2. Submit form
  3. Show the welcome page or redisplay the form with an error message.

You’d probably test it out in IE and Netscape, so that’s three items to check in two browsers. You need six human operations to verify the login page. Pretty manageable to do each time you update the site.

Fast forward to the present, and things get a little more involved.

Sign In - 2010

Today’s login screens are fancy compared to their predecessors. They have javascript validation, fancy tooltips, and other fun stuff:


Counting features on this hypothetical page:

  1. Show the form
  2. “forgot password” link
  3. validate input as they’re entered
  4. show styled error messages for invalid input before submission
  5. a “remember me” checkbox
  6. submit form
  7. Show the welcome page or redisplay the form with an error message.

That’s almost triple the features for the simple login page. Now testing this is more involved, especially since you’re checking it in mulitple versions of a browser:

  1. IE 6
  2. IE 7
  3. IE 8
  4. Firefox 2
  5. Firefox 3
  6. Firefix 3.5
  7. Safari 3
  8. Safari 4

So that’s 7 features over 8 browsers, or 56 verifications. That’s a lot to cover when you’re just trying to get out a damn release (If you’re uncomfortable with IE6 in there, feel free to swap in Chrome).

The login screen is likely one of the least complex pages of your site. If you can’t manage that simple scenario, you’ve got zero chance on the fancier pages.

At this point one option is to give up and leave it to QA staff to hunt down all the browser-compatibility issues. Weak. Totally weak. We’re supposed to be awesome developers. It does little for our reputation when we intentionally pass off unfinished work as “dev complete”.

The other option is to find a way to speed up the testing of the different combinations, so that we’re confident that the delivered site functions across all supported browsers.

Let the Machines Do The Work

Automated browser testing systems to the rescue. These things continuously verify your features. It’s like legalized slave labor. The machine will never get bored or tired. It doesn’t skip test to get out early on a Friday afternoon. It will happily check and recheck your pages all day, even weekends.

This doesn’t mean it’s easy. Setting up good browser tests takes time, and the curve is steep. You may start out with only a handful, like “does the homepage show up?”, or “can users sign in?”. As time goes on, you can create more sophisticated tests until you’ve got a well-oiled verification machine going.

I’ve found that the old 80/20 rule really kicks in here. With only a few tests, you can find and fix tons of problems before they get into the test cycle.

You Have Options

There are plenty of automated browser testing getups out there. I’ve had experience with Selenium. There’s also Windmill (Python), Watir (Ruby), and HtmlUnit (Java).

Pick one and try it out. I’ve found that they’re easier to get running than you might think, and the payoff is awesome, plus it’s fun to sit back and watch a magical browser window fire up and click away. Kind of a Sorcerer’s Apprentice thing (just without all the ensuing mayhem).

(mushroom cloud photo courtesy of giginger, some rights reserved ).

blog comments powered by Disqus