Rust Valley

Technical people complain about tech reporters.  They gloss over details.  They ask dumb questions.  They don’t understand why we are the most important people they’ve ever talked to.

Some of those criticisms are baseless.  Technical reporters have a really hard job. They write about complicated subjects for a really broad audience, so they have to try to both get it right and make it interesting (or at least comprehensible).  The sad fact is that most tech journalists fail on both accounts. This happens in part because tech journalists just don’t have the serious technology background that they probably need to write accurately and coherently about the things they write about.  And it also happens because in an effort to be more comprensible and relevant, many journalists gloss over or flat out change details that matter.

A funny, and frustrating, example of this occurred recently when Jason Perlow tried to explain why Limelight Networks was better than Akamai at serving Olympics content for NBC. Perlow stacks generalization upon inaccuracy and ends up mistating the facts and miseducating readers.  All of this can be clearly, and simply detangled.

Perlow is trying to explain why NBC chose Limelight Networks (Autonomous System Number 22822) over Akamai Technologies (ASN 12222 and 20940 among others) to serve streaming content of the Olympics.  Perlow writes:

“Akamai uses a centralized data hosting infrastructure with big Internet pipes that mirrors content that is hosted on a customer’s own servers. Usually with the aid of a special caching appliance installed at the customer’s ISP or edge network, the request to download that content is re-directed to Akamai’s own servers and fat Internet pipes. When you download big ISO CD and DVD images from MSDN, its going right to Akamai’s data centers over the public Internet. As fat as Akamai’s pipes are, I’ve seen MSDN’s downloads slow to a crawl during peak download periods, such as the days following Windows XP SP3 and Windows Vista SP1’s release. So like my colleagues here at ZDNet, I was expecting the worst.”

So far, Perlow is not even wrong, a phrase coined by Wolfgang Pauli, but which I heard first from Thomas Pogge, a stellar and compelling philosopher who tried valiantly to teach me some Kant long, long ago. That is to say, Perlow isn’t making enough sense yet to be clearly refutable, since it’s unclear what he could possibly mean.  The gulf between the truth and what he writes is so large that the brain of any reasonbly technical person struggles to reinterpret everything he writes in some way that might be closer to reality.

Akamai most certainly does not have a “centralized data hosting infrastructure”.  In fact, Akamai has one of the mostly widely distributed hosting infrastructures (probably the most widely distributed hosting infrastructure) on the planet. Akamai grew up during the go-go days of the late 1990s when space and power were cheap and bandwidth was expensive.  Akamai pitched Internet Service Providers (IPSs) on a value proposition of allowing Akamai to put servers in your space and in exchange serving cached/distributed content to your customers (thereby saving bandwidth).

When I was Chief Technology Officer at Oso Grande Technologies, we ultimately cut a deal with Akamai to host servers with us because we thought it would save us bandwidth.  It did.  Hundreds of other ISPs made the same decision for the same reason. According to Akamai, “Akamai has the most pervasive platform for content delivery and application acceleration – 34000 servers in 70 countries within nearly 950 networks.”  That’s not centralized, hoss.

Limelight, on the other hand, tends to pay for colocation and put their servers in a relatively smaller number of locations (although still a massive number of servers).  This difference in architecture is probably for at least two reasons.  The first is due to when they scaled out.  While Akamai was building out a Content Distribution/Delivery Network (CDN) in the late 1990s and early 2000s, Limelight was still just a Phoenix-based ISP who mostly just resold Global Crossing bandwidth.  Limelight didn’t find their true identity as a CDN until much later.  So they were faced with a very different economic and industry landscape.  By the time Limelight was building out latency was lower overall, speeds were higher, bandwidth was cheaper and power more expensive.

The second reason is that Limelight specializes in streaming large files:  videos and downloads.  These files are much less sensitive to latency than the small picture and html files that Akamai was originally architected for and may also benefit from centralization of storage (to improve cache-locality). That last is a slightly subtle point and if it isn’t obvious to you, ignore it.  Doesn’t matter.  Anyway…

Perlow goes on to write:

Where Limelight differs from Akamai and why the Internet didn’t “melt” is quite simple — they are completely “off the cloud”.  In other words, unlike Akamai and similar content caching providers, their system isn’t deployed over the public Internet.

There are many differences between Limelight and Akamai. Some of those differences are even relevant to performance and scale.  Not only is that difference not related to performance or scale, it isn’t even completely true. Let’s review what he might be talking about.

CDNs replicate content “closer” to users and serve that content up quickly—more quickly than the originator could have served it to that user.  CDNs serve that content on the Internet, of course, so obviously any CDN (including Limelight) is “deployed over the public Internet”.  It would be idiocy to suggest otherwise.

What Perlow might mean is that Limelight replicates to each cluster over their backbone whereas Akamai replicates over the Internet.  That much is true.  He got something almost right!  Akamai does not have a backbone and rather they rely on their colocation and ISP partners (and transit-providers’) bandwidth to replicate content to their server clusters.  Who cares?  In order for that to matter you would have to believe in a world where the first copy of the source content to the server cluster was routinely failing.  That doesn’t make any sense.

The source content is copied out to many, many server cluster who then serve that content hundreds of thousands of times to customers.  Those server clusters are heavy on the outbound-bandwidth and very, very light on inbound bandwidth, since the bits going out go to customers and the bits coming in go to the servers.  Since data circuits on the Internet (except for at your house) are symmetrical, there is always a surfeit of inbound bandwidth available.  Whether you copy the data across a backbone or across the Internet, it’s going to get there.  This difference between the two companies has nothing to do with whether end-user performance is slow or fast.

In the end, we’re left to wonder:  does Perlow understand anything about Internet routing and CDNs at all, or did he just confuse users by glossing over details that he actually does understand?  At this point, we don’t know…

[Disclaimer:  I know and have long relationships with people from both Akamai and Limelight in my day job at Renesys.  They're both good companies doing interesting work.  I don't have a particular opinion over which CDN is better since I use neither.  But both deserve more accurate coverage than this.]