15KB of Fame

When Sydney is closer than Sao Paulo

It's quite possible to make your living from the internet without really considering how it's constructed. I came across this talking with my friend Aaron. He's a bright guy but like many people, assumes the net is done "with satellites or whatever", or never think about it.

The physical structure of the internet very similar to the global air network. Picture those glowing arcs connecting cities on the back of airline magazines. Then recall the hour you spent in line before boarding, the hour spent driving to the airport, the two hours from the big hub airport to the smaller city you really want to go to, and how relieved you are to be traveling now instead of last week when a storm in Chicago somehow messed up flights to Los Angeles. That's basically how your data feels, too.

The long-haul internet is, in fact, a series of tubes. Very fragile tubes.

The long-haul cables that run the global economy are less than 2 inches in diameter, buried under railroad tracks, highways, or ocean sediment. Within densely-settled areas the network is relatively redundant. But between countries and across oceans, data flows through an uncomfortably small set of bottlenecks.

Any minor disaster can damage large portions of the 'net. I recall one incident in 2004 when Miami/Sao Paulo traffic was suddenly re-routed through Washington DC, then New York, then across the ocean to Brussels(!), then back to SP. As late as 1998, a train wreck or wildfire in northern Florida could cut off the whole peninsula for days. This year one of the most exposed points, Suez, suffered multiple failures. The re-routing flooded the already overloaded Europe-Asia network for several weeks.

Even on a good day you can see the problem. Look at how a packet of data might travel from San Francisco to Hong Kong:

(start)
Folsom St, San Francisco (1 mile)
Pine St, San Francisco (2 miles)
Pine St, San Francisco
Oakland, CA (10 miles)
Sacramento, CA (80 miles)
San Jose, CA (120 miles)
Oakland, CA (40 miles)
San Jose, CA (40 miles)
Hong Kong (7,000 miles)
(finish)

That poor little packet of data rattled all around California, looking for an uncongested cable over the Pacific. For each hop, a decision is made to send it on to some other place that may have better luck. The system works pretty well under stress. The point is that the stress is there all of the time.

The long-haul internet is a map of trade volume between cities, that lags up to 20 years behind reality.

This is actually true for all forms of high-volume transport, so there is a lot of history to learn from. Infrastructure is insanely expensive and slow to build even though it almost always pays off in the long run. Short hops between financial/military/political/industrial hubs tend to get built up first. Just look at how many ways there are to travel between New York, Boston, and DC, for example.

Or look at area codes. At the time the precursor to the modern phone system was built, dialing a 1 took 1/9th the time of dialing a 9. Silly, but true. So there was a premium on lower numbers. New York's area code was 212, DC 202, Los Angeles 310, Chicago 312. El Paso, Texas? 915. Anchorage, Alaska? 907. Miami was definitely not a hub at the time, but it was important to the Navy and Air Force. Miami's code is 305.

So if several factors of demography, geography, and politics align, there may form a route of sufficient capacity between two points. If not, too bad. It takes years to build up demand, more years to begin the project, and more and more years to finish it. There are bribes, labor riots, sabotage, political chicanery, etc, and that's when it's a good idea. The first trans-continental rail link in the US was completed in 1869. Twenty years earlier, people had been hijacking ships in Louisiana to sail around South America and crash-land ashore at San Francisco. Other people walked.


Almost as often you'll see huge infrastructure built for political reasons. For example, there is a highway that runs directly from Orlando's airport to Disney World and Cape Canaveral. In those cases trade can grow because the infrastructure is already there.

So what does this have to do with the internet? Quite a lot. The same drama plays out when cables are planned and laid. The connection between Seattle and Tokyo is excellent. Ciudad Mexico and Dallas? Fairly new and really fast. New York to London? World-class. But try to get an email from Barcelona to Bangalore, and often you'll find that it routes through America. Companies are scrambling to build up Europe-Asia links. They've been scrambling since the late 1980's, and it will be some time before they get there.

The long-haul internet is not a magic leprechaun.

Let's imagine a perfectly balanced world-wide network. You have a business based in San Francisco. Your people are in SF, your technology suppliers are in SF, most of your customers are in the United States. You have a small but growing customer base in Hong Kong and China. We have a perfect 'net, so there are no silly congestion problems and there is just as much bandwidth across oceans as between cities. Rack space in HK is twice the price as in SF.

So what is the no-brainer place to put your next server farm, hire people to maintain it, set up office space, pay property and business taxes, etc? Correct. Hong Kong. No matter how good the internet gets it will never be faster than a small fraction of the speed of light.

Recently I was looking at the server logs of a site located in the US. The response times had a very high variance, which indicates a severe bottleneck somewhere. After a lot of poking around, trying to find the problem within the server farm, I had the bright idea to segment the logs by source country. And there it was: the response times were all over the map because the users were all over the map. From the perspective of the 'net, Stockholm and Singapore are just down the block while India and Sao Paulo are past the moon.

Rules of thumb:
- Light takes about 100 milliseconds to travel 10,000 miles and back.

- A network packet on a good route may take 3 to 10 times that long.

- A network packet has not "arrived" until a return acknowledgment has been sent all the way back.

- The longer and more complicated the route from here to there, the higher the chance (often more than 10%!) that a packet will get lost.

- The larger the file you send, the more packets it has to be chopped up into, the more likely one will be lost and have to be re-transmitted. All else being equal the transit time of a file increases more than linearly to its size.

Once upon a time, Adrian Thompson accidentally created a magical artifact. He used a genetic algorithm to evolve a FPGA circuit that would distinguish between signals of two frequencies. The winning specimen was pretty unlikely: it worked on that chip only, and only at certain temperatures. It contained circuits that were not connected to the others, but when he turned those off the damned thing stopped working. The neighboring gates depended on that extra bit of inductance, which varied with a slight flaw in another gate... maybe. No one really knows.

Once upon a time, a small group of hackers evolved a piece of software. They focused so much on doing more with less that they themselves became crucial parts of the implementation: Leo took over ops because he stayed late to avoid traffic anyway. These support emails routed to Bianca because she spoke Spanish and Portuguese. Tatiana had a nicer touch with angry customers. Carlos had the pager because he's good at troubleshooting. Tom made the changes to the template parser because Tom's the genius who wrote it. The bug tracking system was a whiteboard. Every little detail seemed to depend on some idiosyncrasy of the people or their environment. It worked -- somehow. Every startup is a freak accident.

This is why I think Jaron Lanier's bomb-throwing about closed- versus open-source focuses on the wrong thing. When you're talking about "radical creativity" in a program, it doesn't matter how many people are allowed to read the source code, but how many are required to understand it. Having too many junior hackers on your source tree is just as bad as too many customers pestering you about compatability.


The fundamental difference between the first two systems and the ones that come out of large organizations is that there was no difference between design and implementation. Enterprise code is designed, vetted, reviewed, implemented, deployed, life-cycled, etc. The purpose is to have no irreplaceable parts. On the other hand, the magic chip and the startup were evolved as fast as possible in a way that could never be exactly repeated.

When an enterprizey programmer and an alpha-hacker look at each other's code, the same thought occurs: "This is a rambling mess! How do they get away with this crap?" The documentation is the code, alpha-hackers say. Don't you see that you're writing your programs in PowerPoint? The process gurus grumble about "heroics", maintainability, knowledge transfer, bus accidents. The enterprizey people regard design as something that must be as separate as possible from the expression-as-code. How else can it be maintained by anyone? Take away Tom and the thing no longer works. True, but if enterprizey was always better, BigCo wouldn't get their lunch eaten by wittle bitty startups.

It gets more fun when the big ones turn around and eat the little ones. Most startup acquisitions play out like a spider dissecting a fly: gruesomely impersonal and tedious. You just can't get over the image of that fat spider pick pick picking away at a violated husk. Yahoo's epic rewrite of RTML, or its painful digestion of my company's system, are the same phenomenon as that spider, or that poor bastard trying to understand his magical chip. There are bits that work only there & then, bits that may be impossible for an outsider to grok completely, bits that even the same people would do differently now.

At least 50.7% of the voters in Venezuela would rather not give their president dictatorial powers and a shot at life in office.

The "socialist paradise" bribe in the constitutional referendum was transparent: a 6-hour day and some worker's benefits in exchange for astonishing new powers for Chavez, including no term limit and an extension of the term from 6 to 7 years. The 7-year thing really bothered me. The only reason I can think for it is that, being a prime number, a 7-year term would put him out of step. At the times most other leaders would be sweating re-election he would have lots of elbow room.

But Chavez was out of step on the economy. The bribes were to be funded by oil exports, which are strongly linked to the dollar. With the dollar going down, there is enormous inflation pressure in oil-producing countries. Chavez' answer to that was price and currency controls. The official rate is 2,200-odd Bolivares to 1 Dollar, with tight limits on how much you can buy or sell. As of last week the black-market rate is over 5,000:1. To wish inflation away he put in price controls on basic foodstuffs, with the unsurprising result that there is less and less stuff on the shelves.

Just as Solidarity in Poland was partly fueled by the price of sausage, the NO campaign got boost from the scarcity of Doña Arepa. You can shut down the TV stations, you can shoot the protesters are arrest the "traitors", but you can't monkey with the price of milk.