Without measurement, organizations that rely on the Internet for all or part of their business cannot identify problem areas, monitor user compliance with service-level agreements or fully understand users’ online experience. Keynote Systems has been measuring Web performance since 1997, and in that time it has seen Web sites grow in complexity and value.
The San Mateo, California-based company also has seen its own value grow. Keynote garnered revenue of US$9.7 million in its first fiscal quarter of 2004, an increase of 2 percent versus the prior quarter. The company’s traditional performance-measurement services continued to stabilize, while its performance-management offerings grew 23 percent compared with the same quarter in the previous fiscal year, according to Umang Gupta, Keynote’s chairman and CEO.
Furthering its emphasis on management and services, late last year Keynote acquired the Insight service suite from the Xaffire division of MatrixNet Systems. The company immediately began offering that Internet diagnostic service under the name Keynote Network Perspective.
Lloyd Taylor, vice president of technology and operations at Keynote, recently spent some time talking with the E-Commerce Times about the Web performance and management markets, Keynote’s work in these areas, and which trends e-business executives should track in coming months. Taylor, who joined the company in 1999, is the chief architect of Keynote’s measurement infrastructure. He also manages Keynote’s operations, customer service and business systems group, in addition to spearheading the company’s Red Alert division.
E-Commerce Times: How long has Keynote been measuring performance, and how has this area of focus changed in the past few years?
Lloyd Taylor: Our first measurements were in ’97 — simply the measurement of the performance of the main page. Initially, it took about 13.5 seconds to download. Nowadays, it’s more like eight-tenths of a second. The next thing was transactions, which were multistep page views. That added more complexity — buying a share of stock can take you through five pages.
Then we started measuring streaming performance as more and more sites used video and audio, especially as people started selling online. More recently, we’ve seen a lot of interest in wireless performance — accessing the Internet with a cell phone or a PDA or BlackBerry. We’re offering these measurements as well. As with any other media, it starts with the simple, then grows in complexity.
ECT: And where is the most interest these days?
Lloyd: The most interest is not just in what’s happening, but in modeling and measuring performance to see what the limits are. There’s a lot more interest in testing from the perspective of the end user. If you test 5,000 customers who happen to be internal, you’ll have great performance. Load-testing has to be done from the outside.
ECT: How are some of the tools or services you’re using to provide external testing evolving?
Lloyd: There is no longer a one-size-fits-all [model] for business. We’ve added things like measuring each piece in the transaction line so it gives an absolute representation of the user experience. It’s for people who want high-fidelity measurements. We have other products that don’t measure high fidelity but cost less to run — that won’t interpret Flash, for example, or execute Java.
We can actually replay a user’s experience, just like a TiVo. It’s done with a single code change on the home page. Everything is easy, low-hassle. It’s a service model, a portal base and very easy to use.
ECT: And how does the service model compare, price-wise, with purchasing the software?
Lloyd: If you look at some of the software products, it can cost $1 million or $2 million to license these things, then you need five or six people in your company who know how to use these tools. If you’re doing load-testing once a quarter or once a month, it’s not worth it. Buying software is expensive, and keeping it running is even more expensive.
ECT: Is Web performance a hardware issue? A software issue?
Lloyd: It’s really neither. It’s really people and philosophy. The fastest Web sites we measure are Web sites like Google and Yahoo. Now why are they fast? Ultimately, they’re fast because the designers have said, “We want to have the minimal content possible that makes our customers happy.” So look at the Google home page. Would you imagine that a company that has — whose book value is what, $20 billion or $30 billion — would have such a sparse home page? And yet they very carefully thought about their customer base and what their customer base wanted. And what they wanted was very fast response. The speed really has to do with what you’re trying to do and who you’re trying to please.
That said, once you’ve decided whether you’re a content-light site or a content-heavy site, then yes, absolutely, your design of hardware, your design of software, does your site involve databases, [all become factors].
Business should drive technology. One of the interesting outcomes of the last three years has been a lot of business is now in charge of technology, whereas five or 10 years before that, technology was driving business. That was a good thing because it blasted business out of its old way of thinking. Now what’s happening is businesses are saying, “Great, we bought a load of new stuff. Now let’s make it financially sensible.”
ECT: Do the various operating systems have any effect on speed or performance?
Lloyd: I think it goes back to the design and the willingness of the designer to implement the best tool for what they’re trying to do. The biggest problem you’ll see, in my opinion, from the technology side of the Net, is that a lot of the folks in implementation tend to be technology bigots. That is, “I will never, ever use Windows,” or, “I will never, ever use Solaris,” or, “I will never, ever use Linux.” Those are just tools. It’s kind of like saying, “Well, I have to drive a screw, but I’ll only use a hammer.” I mean, you can drive screws with hammers, but it’s kind of hard on the screw and it doesn’t do a very good job.
Whatever it takes, it should be a systems approach to thinking about what the business does — what am I trying to achieve, what are my metrics? And here’s a critical one: How am I going to measure whether I’m successful or not? That then drives how you design, implement and measure what you’re doing.
An interesting factoid for you: Keynote does a lot of indices now in measuring a selected basket of Web sites. Without exception, within six months of Keynote starting to publish a metric on that basket of Web sites, performance improves by at least 50 percent. In other words, when we start measuring something it might be 13 seconds, but within six to nine months it’ll be running at six seconds. Why is that? We’re finally making visible, in a comparative way, how well a site performs. People say we’re just trying to get marketing hype. Actually, this really improves things for the consumer.
Think about the best Web sites you use every day. What is common among them? You might be someone who likes content, who likes everything right there. You might be willing to wait a couple of extra seconds. Or you might want to get your information quickly.
ECT: What are companies doing — once you’ve measured their sites — to improve their performance?
Lloyd: Again, the key thing is understanding what it is you’re trying to improve. The classic problem is the army’s only fighting the last war. Well, the implementers of Web sites are only fixing the last problem. So for a long time, people would just throw hardware at things.
Let me just give you a for-instance. I can’t name the customer, obviously, but there’s a load test we did for a major customer a couple of years ago, and they were being told by their outsourced hardware-software-services vendor that the site was slow and it was going to cost them $2 million to upgrade the hardware and software. We did a load test, for which we charged about $50,000, and what we discovered was there was a misconfiguration in one of the software characteristics of their Web server. With the correct configuration, performance was even better than they’d wanted. We probably pissed off the hardware-software-services vendor. But on the other hand, the customer saved $2 million minus $50,000.
So step number one is to figure out what your real problem is. And a common problem there is if you ask whoever did the implementation, they’re going to be focused on their piece of it. The first thing to do is go to a third party that has no vested interest in your hardware or your software or anything, and say, “Do an audit to find out where the problems are.” This is a powerful tool, not only to help that department figure out what has to be done, but to explain to the CIO what it involves and what improvements it will create.
ECT: Is it typically a misconfiguration or something like that?
Lloyd: In a lot of cases what you’ll see is it was a decision by the content folks. I remember we were doing the last presidential election. We were measuring the various Web sites of the candidates, and there was one Web site from — again, I won’t name names — which was abysmally slow. The reason for it was they had a little 1.5-inch by 1.5-inch picture of the candidate, but they hadn’t tuned that picture. It was a 1.5-megabyte picture that was being dynamically downsized to 1.5 inches by the browser. If they’d taken two seconds with Photoshop or something like that and stripped it down to a 10-KB image, it would have looked just as good and the Web site would have been really fast. If you go to the systems administrator and say, “Why is my Web site so slow?” that person is probably not going to spend a lot of time looking at the content.
You really need to engage someone who can understand the whole system — not just the hardware and software, but the Internet connections, how content works, those kinds of things.
ECT: Once they’ve done that, how easy is it to fix the problem?
Lloyd: It needs to be a business process as opposed to just a technology process. It’s very easy to spend tons of money on hardware, software, consulting services, but almost every time … money [is] used in less than optimal ways.
For example, if you have a 200-KB main page and you want to double performance, you have two choices. You can either double the hardware and the pipes and everything else, or you can figure out how to reduce the main page to 100 KB. It may be that you decide your customer base really needs that 200 KB page, in which case, it’s fine. It’s a business decision. Go buy the hardware. Go buy the bandwidth. Or you can say, “I’d rather outsource the heavy-bandwidth stuff to a CDM.” That may be a good business decision. Again, it’s a systems approach. It’s very hard to find people who will help you at a systems level. It’s not something that happens much in any kind of industry.
[We give clients] a list of things you can do, easily and inexpensively, to help reach your goal. And then [we] say, “Here’s where you’ll have to spend money.”
The second part of that is, once you’ve made the improvements, you have to have monitoring. If you aren’t measuring it, you aren’t managing it. It’s like, when you’re sick, going to the doctor to get well but then not watching your weight, watching what you eat, getting a check-up. Those are the kinds of things we do in our measurement practice — keep continual track of how you’re performing. A number of customers are using our measurements for enforcing service-level objectives with their providers. They make back 10 or 20 times what they spend on us.
ECT: Is the increasing adoption of high-bandwidth access changing Web design and Web performance?
Lloyd: I think for those that are specifically targeting individual users, it certainly is leading them to include more content [and] make more streaming available. There are whole new business opportunities opening up that way — iTunes, Napster — but, again, it really depends on who you’re targeting. To what extent is broadband demand driving that, or to what extent is it just a natural progression? You just keep adding more and more until somebody screams.
ECT: Is caching having any impact?
Lloyd: Content distribution managers — CDMs — are now a pretty mature industry in their main product. They’ve been very, very important in areas such as the antivirus world. Pretty much all the antivirus vendors download and stream using CDM. They’re also very effective in areas where there’s a flash crowd possibility. A flash crowd I define as a sudden interest in a Web site for an unpredictable reason. For example … September 11th was the first time we saw the public significantly turning to the Internet for their news, so all the news sites were hammered. CNN when September 11th happened was very smart. They very quickly struck their page down from .25 megabyte to about 10 KB, which gave them a 25-fold improvement in throughput.
The next time we saw the same thing was a few months later when Bush was scheduled to declare war. At 4:00 Eastern time, the Web sites were completely inundated, but because they were using CDM and because they had their plans in place, the content got out.
The other thing where it can be really effective is actually personalizing sites. What you’re seeing more and more is the Web site of a large provider customized for specific regions. That customization’s being handled by CDMs on the fly.
ECT: Will IPv6 alter Web performance at all?
Lloyd: I don’t think it’s really material. For one thing, it’s going to be transparent to the majority of end users, assuming it ever happens. The difference between the academic Internet — how it should work — and the real Internet — how it does work — is the academic Internet is always focused on what’s best, whereas the real Internet or the commercial Internet is based on what works and what works well enough.
Whenever you look at a new technology, you have to ask, “Who’s going to benefit from it? Who’s going to pay for it?” Well, we’re not running out of IP addresses. The typical corporation only needs 200 or 250 addresses. There’s no motivation that we’re going to run out of IP space. The thing I see driving it will be stopping spam, quite frankly. Anybody using IP version 6 will have a hard time hiding their identity. In a lot of ways it’s a solution in search of a problem. Now, I may be wrong, but no one has been able to sit down and explain to me the economic benefits of making the change. And, as interesting as it is, it needs economic benefit for it to happen.
There is one exception, by the way: cell phones. Cell phones are heavily implementing IPv6 — in Japan, anyway. It may actually be the wireless networks that drive IPv6, and those are all low bandwidth.
ECT: Is there something missing from the area of Web performance?
Lloyd: The big hole really here is on the owner of the Web site. Anybody who’s making money off the Net or keeping people happy off the Net should be measuring and managing from the outside. A lot of them are not. And you end up with a situation where you have unhappy customers that you don’t know about.
The second area where people are not paying enough attention, in my opinion, is the user experience. There are a lot of really badly designed Web sites out there that are really hard to use. Spending a little bit of money on something like our WebEffective product to actually see what users are doing and where they’re getting frustrated is probably one of the most underutilized [options]out there now. We use it on our own site, and it’s really stunning. As we view how people use both our site and our customer portal site, we’re amazed that we’ve optimized for the wrong things. We do this on a quarterly basis because customers’ needs change.
A lot of the sites that do it well are some of the airline sites, certainly the travel sites. They’ve had to pay attention because they compete with each other on how well designed the sites are. To pick one out of the air, as a personal example, look at Northwest Airlines. What you see more and more is one-click stuff going to the main page.
The first generation of performance was: How fast is my site, my main page? The second generation was: How long does it take a customer to complete whatever they want to do on my page, what we call transactions? That included a little bit of usability because if it took 16 pages to buy a share of stock as opposed to four, the 16-page trading site came up with a poorer number. So they became more efficient.
I think we’re now in the third generation, which is recognizing that performance is a function not only of your hardware, your software, your network, but also of what your customer’s trying to do. This generation, in my mind, is optimizing customer performance. In some ways, it is actually easier to do that than it is to build a more powerful Web server to optimize your database. Stop getting in your customer’s way.
Loading ...
Social Media
See all Social Media