Why Johnny Can’t Read Your Web Page (and How To Make Sure He Can)

Web pages that render properly in some browsers and not in others are a big headache for site managers – and a pet peeve of mine. In a perfect world, pages would look and work the same in all major browsers; of course, in the real world, that simply just doesn’t happen. To uncover these flaws before users do, cross-platform and cross-browser testing is customarily performed before the new pages go live. Too often, though, it is a step that gets less rigorous attention than it should.

With many years of experience under my belt, I take it with a grain of salt whenever a member of my development team provides me with completed web code and assures me that ‘it works’. I begin by opening the site in Safari (my preferred Mac browser) or Firefox, both of which seem to handle standards-based web code well. Sure enough, the page nearly always renders perfectly. So far, so good. Tempting fate, I turn my attention to testing it in Internet Explorer 7 or 8, the preferred browser of the majority of web users.

Then everything goes to hell: font’s don’t look right, the spacing of text might be off or items are completely misplaced on the page. It almost never fails.  Why is it that a finished product that is said to work so often doesn’t?

Frequently, it’s simply because someone didn’t test in IE7 and IE8. This is often because the majority of developers prefer Firefox or some other more rigorously standards-based alternative instead of IE as their default browser. When testing, they will do what most people do: open the page in their default browser. Unfortunately, this browser happens not to be IE. A second common explanation is that the developer might be creating the page on a Linux machine, where Internet Explorer is not available, where they can’t test in IE. (One could argue that they should find a way of testing in IE. While I agree with that, I’m merely stating how thing are.)  In short, developers are creating and testing websites using an environment very different than that used by ordinary users.

This pleasing but simplistic explanation is satisfying but it ignores the complexities that surround website development and testing. Sometimes things that should work – things you wouldn’t even think to test – will end up breaking. Have a look at the graphics shown below. They show the upper-right portion of a client’s website, rendered with the same version of Safari (4.0.5) on a Mac(top) and a PC(bottom). Despite using the precisely the same browser, there is still a difference in appearance – an ugly one. It would not have occurred to me that this could happen, but perhaps I should know better. (Fortunately for me, on this site PC Safari users constitute a very small minority.)

The Scope of the Problem

When you consider that there are several major platforms (PC, Mac and Linux), many different browsers and plenty of different versions of each, the problem of testing them all quickly escalates out of control. But how big a problem is it, really?

For the moment, lets assume I’ve hired the quintessential Rock Star Developer, the sort of person who designs, codes and tests like nobody’s business. In order this person to deliver a finished product that works correctly on any browser, how many different cross-platform, cross-browser testing scenarios would be needed?

At the outset, it’s a given that testing needs to encompass the four major browsers for both PCs and Macs. These browsers would be – in rough order of popularity – Internet Explorer, Firefox, Safari and Chrome. And because not everyone is always using the latest version of their browser and different versions of the same browser can render pages slightly differently, you’ll want to test at least the most current release and the last major release. That doubles the number of scenarios to eight. Then you must factor in two major hardware platforms – PCs and Macs – which further increases the number of test scenarios to 14: eight on the PC and six on the Mac (IE hasn’t been available for Macs in years).

Without any further analysis, you could be reasonably sure that these fourteen test scenarios would ensure compatibility with the vast majority of browsers that visit your site. If you wanted to be even more rigorous you could test some of the lesser used browsers, such as Opera or those popular on mobile devices. And if you’re looking to be really thorough, you could test on different versions of the same platform, say Windows 7, Windows Vista and Windows XP. Add it all up and you could easily be looking at dozens of testing scenarios – and each time you find and fix a bug, you need to go back and re-check all those same variations.

The average site manager doesn’t have the time or resources to test all those combinations, nor does the developer. But that is, in fact, what it takes to ensure that things work properly for the majority of users, that is what our fictional Rock Star developer is up against.

It seems a daunting task; impossible, really. Fortunately, there are techniques to help intelligently narrow down the list of test scenarios to something more manageable, as well as tools to help make the job much easier.

Deciding What To Test

Let’s immediately toss aside the notion that testing dozens of combinations of platform/browser/browser-version is either practical or possible. It isn’t. Given that, how can you know what should be tested?

The obvious path of least resistance is to use the test scenarios I previously mentioned, which means you’re going to have 14 different variations. That’s still an unappealing amount of work. Of course, you could decide, for instance, not to test Mac combinations and save yourself about half that work; but if you do you’re going to have about 10%-15% of your users unhappy when things don’t look or work right. Instead of that, could decide to test only the most recent version of each of the major browsers on each platform. This would also cut your work in half, but then you’d be leaving yourself open to an even bigger audience of potentially unhappy visitors: the ones using slightly older browsers that may not render pages quite the same way.

But why guess when there is a place you can look that will tell you precisely what should be tested?

The perfect test plan is no more than a few clicks away in your web analytics data. There, in addition to the prominent stats like page hits and referrers, you’ll find all sort of interesting data about your site’s visitors, including what browsers, platforms and even screen sizes they prefer. (If you’re not running web analytics you’re flying blind. Stop right here and get a free account with Google Analytics or Clicky and start measuring your traffic)

In Google Analytics, the place to find this data is the Visitors -> Browser Capabilities -> Browsers and OS section. Clicking it will bring up a handy list of the types of browsers being used by the visitors to your site and the percentage of visits associated with that browser.

These results tell you precisely what you need to know. On this particular site, I can see that if I support two versions (current and most recent) of IE, Firefox and Chrome on Windows, and Safari on the Mac – eight testing scenarios – I should be able to guarantee nearly 94% of my visitors. Eight test combinations is certainly much less work than the original 14. (Note: If I add Firefox for the Mac, I can get to a bit over 96% with just one additional test variation. Finally, if I test Safari on the iPhone, my 10th test, I can guarantee that 97% of the users – representing the majority of my mobile users – will be happy.)

Bear in mind that the summary numbers shown in that chart don’t tell the entire story. Drilling down on each browser reveals a more nuanced picture. For example, clicking on the first entry, Internet Explorer, reveals the following details:

Here the chart is saying that of those IE users, the ones that account for nearly 53% of this site’s traffic, more than half are using IE8 and slightly over a third use IE7. (IE6 brings up the rear, but it’s such a terrible browser that few sites actually support it anymore) So, my original contention – test the most recent release of the browser and the one prior – remains true.

Now lets take a look at the same chart for Firefox:

This one’s a bit harder to parse. Unlike IE, which simply lists major releases (5, 6, 7 and 8), Firefox lists minor releases, which means there are more potential variants to test. My advice here is only slightly different from what I recommend with IE: test the most recent version (3.6.2) of the latest mainline release (3.6) and the most used version of the previous mainline release (3.5.8)It’s cutting a few corners, but you should be able to get away with it. (No guarantees, though)

To summarize, by using Google Analytics data, I’ve determined that my test plan for this particular site means the following combinations:

• IE 7 and 8 under Windows (Vista, 7 or XP)
• Firefox 3.6.2 and 3.5.8, also under Windows
• Safari 4.05 (the latest) and 3 under Macintosh
• Chrome 5 and 4 under Windows

That’s eight different test scenarios. Now, how do we do it?

Tools

Having numerous machines, each configured with the appropriate operating system and browser combination – the brute force approach – is what springs to mind immediately. Of course, this is both ridiculous and impractical. It’s absurd to contemplate that many machines dedicated to nothing but testing. (To properly test you should be running machines in their unmodified, default configurations, without 3rd-party plugins or application which could potentially poison your test results. If you’re following best practices, that means using these systems for other tasks is pretty much out of the question.) Sure, you won’t need 8 different machines; you could, for example,  simultaneously run IE, Firefox and Chrome on the one system, but you couldn’t run IE7 and IE8 on the same system. The same is true of Safari. So, you’d need at least four computers (two PCs and two Macs) to accomplish your testing. Still wasteful.

A much better solution is to use virtualization technology, which allows you to simulate many different machine environments perfectly on one computer – just cycle between emulation environments to do your testing. Another solution is to use web-based tools, some free and some not, that eliminate the hassles of setting up a complete virtualization environment. Still more tools allow you to take quick screen shots of how the site’s pages render in the browser. In the sections that follow I will discuss each of these solutions and their pros and cons.

Approach: Web-based Snapshot Tools

If you have a site that has static pages with few or no interactive components, such as a blog or a simple sales landing page with a ‘Click Here To Order’ button, you may be able to get away by using web-based snapshot tools. These are a low-tech but can be surprisingly effective. You provide these sites with a URL and they generate screen shots from machines running the actual platform/browser configurations requested. Examining the resulting images will reveal any obvious rendering issues.

These services give a fast, if cursory, view of how things look. They can be helpful when you’re in a pinch and just need a quick reality check on a given browser/platform combination. Just remember they’ll provide no help at all for interactive testing. If you have an e-commerce store or other site that is heavily interactive you still need to perform usability testing.

Here are some the of the solutions available:

• Litmus
• Browsershots
• BrowserCam
• IE NetRenderer

Approach: Virtual Machines

One of the major technology advances of the past decade has been the virtual machine. Virtualization is, simply put, a means of allowing one physical machine to be sub-divided into many smaller ‘virtual machines’ (VMs) each running their own copy of some operating system and each having a portion of the host machine’s memory, disk and network bandwidth. For instance, you could have one powerful physical server that hosts, say,  seven VMs, each running a different version of Linux or Windows. For all practical purposes, VMs work just like real machines.

Virtual machines have been widely adopted in data centers, where they have revolutionized operations by providing a means of making fuller use of machine resources that previously went to waste. With few exceptions, the old way of doing things – having each physical machine dedicated to one major task, like serving web pages or delivering emails – is no longer how things are done, all thanks to virtualization. This blog, in fact, is hosted on a virtual machine that shares resources with seven other virtual machines on its host server. For most sites, virtual hosting is efficient, effective,  economical and completely transparent; so transparent, in fact, that your site may already be hosted on a VM without your having realized it. (See here for a discussion of the benefits of VMs (aka VPS) for site hosting)

While virtual machines are widely used in data centers, the average computer user has little need for them. For those actively working with technology, though, virtual machine technology is tremendously useful. Remember I wrote earlier how impractical it would be to have so many physical machines to perform browser compatibility testing? Well, with VMs it’s not only practical, it’s easy.

On my Intel-based Mac laptop, I’m running Mac OS X with 4G of RAM and Parallels, a virtualization application. On this machine I have a VM running Windows XP with IE7, Firefox 3.5, Safari 3 and Chrome 3; a VM running Windows XP with IE8, Firefox 3.6, Safari 4 and Chrome 4; and another VM running a copy of Ubuntu Linux with the latest version of Firefox (and several other Linux browsers). In addition to these virtual machines, I have my native operating system, Mac OS, running the latest versions of Firefox, Safari and Chrome. With this configuration, I can quickly test more than a dozen different browser/platform combinations on one modestly equipped, three-year old laptop. Unlike the aforementioned screenshot tools, this solution lets me interact with the site I’m testing, giving me assurance that the page is both rendering faithfully and performing as it should interactively.

Here is a screen shot of my Mac running Parallels. On the left is Safari 4.05 for the Mac; on the right, the same page and browser under Windows. Cross-platform testing doesn’t get any easier than this:

Virtual machine technology is available for all the major desktop platforms: Windows, Mac OS and Linux. The major solutions are:

• Windows/Linux: VMware, Parallels
• A variety of free solutions also exist for Linux
• Macintosh: VMWare Fusion, Parallels

Note that the Windows-based VM products can’t run Mac OSX. However, the reverse is not true: the Mac versions do let you run Windows. So, right now, at least, the best way to test all the major combinations of Windows, Mac and Linux browsers in on place is on a Mac.

As with any virtualization environment, the initial setup involves a little time, effort and some expense. You’ll need to purchase the Parallels or VMWare software and you’ll also need Windows licenses. That aside, this is the perhaps the best testing environment imaginable.

Amazon EC2

If you want to avoid the hassles of setting up so many environments and begin testing right away you can still use VMs. The cloud comes to the rescue, in the form of Amazon’s EC2, where, in minutes, you can fire up Windows or Linux virtual machines that will let you get right down to testing. You’ll pay by the hour for using Amazon’s infrastructure, and over the long-haul it may be cheaper to to build your own; but for on-demand access and quick test runs, it simply can’t be beat. It’s also available worldwide, so whether your developers are in India or Idaho,  they can now test their pages properly in IE  before telling you “it works”.

Finally, bear in mind that EC2 only provide Windows and Linux environments. If you need to test Mac compatibility, you still need a real Macintosh.

CrossBrowserTesting.Com

The last option I’ll offer up is one that combines many of the best features cloud-based virtualization along with some screenshot tools. Additionally, it provides Macintosh test environments. In contrast to Amazon’s hourly pricing, CrossBrowserTesting.Com charges a monthly subscription fee which buys you a certain number of online minutes of testing. I have not used this service since, as I described earlier, I run my own VMs; but if you want to get started quickly, need both PC and Mac test environments and don’t want the hassles of setting up your own VMs, this service might be the way to go. A free trial is available.

The Fly in The Ointment: Hand-held Mobile Devices

Apple’s iPhone has led the charge towards mobile devices that are as competent at browsing the net as they are in making phone calls. On the iPhone and smart phones like it, you should be able to open just about any web page and see it rendered faithfully, albeit microscopically. Until now, however, small screen sizes on these devices made for poor usability. Additionally, some page elements, like Flash and Java, didn’t work at all; still other elements were simply unworkable due to the lack of keyboard and mouse. Given these limitations, user expectations were low and few people complained when they went to their favorite site and found it somehow lacking.

And then came The iPad. This device promises to change a lot of things, not the least of which are the aforementioned low user-expectations. With its larger screen and faster processor, people are planning to be able to do things with this device that they simply wouldn’t have even tried with their phones. They’re going to expect to use it almost like a regular desktop and they’re going to expect it to work. That means more testing.

It’s probably too soon to say how popular the iPad is going to be; early indications are: very. And it won’t be just Apple operating in this space. Other manufacturers are surely going to enter the marketplace with similar hand-held devices, and that means even more testing scenarios. In particular, it means placing a greater emphasis on usability testing, because handheld interfaces are much different than the traditional keyboard and mouse we’ve all been using for the last several decades.

Mobile devices can be a bit tricky to test. Many of the cross-platform testing solutions I’ve already outlined won’t work. You can’t, for example, run an iPhone VM to test browser compatibility because there’s no such thing.  Adding to the problem, none of the services I’ve mentioned support mobile devices. So, are we back to square one, do we need to buy each of these mobile devices to test?

Fortunately, the answer is no. While mobile devices can’t be run as standalone VMs, all of the major mobile device manufacturers provide software development kits that include emulators for testing. These emulators can be installed and run on Windows or Mac VMs; just fire up your favorite VM, add the SDK, and test your pages on the mobile device. Apple’s SDK provides both an iPhone and iPad emulator, though to test the latter you’ll have to fork over $99 (the iPhone emulator is free). Android-based devices have their own SDK and emulator (no charge). RIM, the creator Blackberry devices, also offers emulators for their many devices (no charge). While Apple and Android mobile device emulators run on both Windows and Mac, the Blackberry simulators are Windows-only.

By smartly combining the right computer with a virtual machine emulator and the various mobile device SDKs, it is entirely possible to have your own one-ring-to-rule-them-all testing platform, a place where you’ll be able to quickly switch between each of your desktop and mobile test scenarios.

• iPhone/iPad SDK
• Android SDK
• Blackberry Simulators

You Can’t Support ‘em All

I won’t guess, nor would I bother to try counting, the number of browsers out there. In the end I’m simply forced to concede that supporting every one of them through testing is just not feasible. But even so, that doesn’t mean users with untested browsers – the remaining 4% – 6% – should be left to assume your site is broken if it doesn’t render properly in their bleeding edge browser.

It would be much better to inform users when their browser is incompatible and request that they use one that has been tested. The tools and techniques shown below, one of which is a commercial product, let you do exactly that.

If you’re going to fail, fail gracefully.

• BrowserHawk
• BrowserDetect JS
• Apache Browser Detection

Additional Considerations

There are a few additional items I haven’t yet discussed, in part because they may not be applicable to all environments. Nevertheless, they should be considered because they may have some impact on your testing scenarios.  I’ll touch on each briefly.

Flash/Java

Flash is the most widely used means of delivering video on the Internet. It is also used on a great many web pages where you might not realize it, to do things that might otherwise be difficult or impossible in HTML. (These are sometimes referred to as UFOs – Unobtrusive Flash Objects). As such, it is deeply embedded in the Internet. Its lesser cousin, Java, is also used, though certainly much less than  it once was. If you have pages that use either of these technologies, they undoubtedly work just fine in most browsers.

But there’s trouble brewing on the horizon. Apple’s has steadfastly refused to support either technology in any of their mobile devices. Apple’s CEO, Steve Jobs, is not known for his flexibility (see: The 1-Button Mouse)and at this point it seems a safe bet that neither Flash nor Java will ever be allowed. Until the introduction of the iPad, people just accepted the lack of Flash (few seemed to care about Java), albeit grudgingly. But if the iPad takes off as I believe it will, a growing share of web users will be connecting to sites using a device that is unable to completely render pages using either Flash or Java. This means site managers are going to be faced with the unpalatable choice of redesigning pages to remove Flash and Java components in order to be compatible with the latest generation of toys from Cupertino.

Screen Resolution

It’s one of the early decisions that must be made when designing a page: What is the minimum screen resolution this site will support? Of course, the days of 640×480 and 800×600 are long gone, and 1024×768 is fading rapidly, too. So you’re probably designing your sites for larger screens, which is fine. But when it comes time to test these pages, it’s important to remember that you should be testing at the defined minimum necessary screen resolution, just to be sure you don’t end up with any embarrassing horizontal scrollbars at the bottom of your page. Testing at a wider resolution might also be necessary, to see if your layouts are consistently liquid or if some elements remain anchored to the left margin while others float across the page.

Connection Speed

Connection speed is obviously a performance issue, not a visual element. Still, it shouldn’t be ignored during testing because it will affect your page load times. Pages that take a long time to load bore users who won’t wait long before they give up and go elsewhere. Fortunately, most people have, by now, switched away from dialup, so you shouldn’t worry too much about them. (To see how many dialup users hit your site, in Google Analytics, go to Visitors->Network Properties->Connection Speeds) But in the componentized world of web design, where enhancing a page can be a simple as a few lines of Javascript, if you’re not paying attention it’s easy to design pages that load slowly, even on a broadband connection. Worse than presenting a poor user experience, slow page loads can affect your search engine rankings – and then you’ll have fewer poor user experiences to worry about.

Conclusion

Cross-browser, cross-platform testing is a vital element of proper site management. It’s also, I’m sorry to say, a thankless job: do it right and no one notices a thing; do it wrong, and you’ll hear it from your users. Though comprehensive testing of every possible machine/browser permutation is impossible, the techniques I’ve outlined in this article will help you to whittle down the enormous list of potential test scenarios to the relevant few. By adding virtual machines and other tools to your testing regimen, you can create an environment that will allow you to quickly switch between each of your test scenarios, turning a once daunting job into a relatively streamlined process.

Additional Sources:

• 7 Awesome Resources to Test Cross-Browser Compatibility of Your Website
• 7 Fresh and Simple Ways to Test Cross-Browser Compatibility
• 10 Helpful Resources for Cross-Browser Testing
• Useful Tools for Testing Cross Browser Compatibility

Tagged as: Amazon EC2, Chrome, cross-browser testing, Cross-platform testing, Firefox, Internet Explorer, Opera, Safari, Testing