Carlos Bueno

Measuring Javascript Parse and Load

2010-02-10T15:12:00.000-08:00

Any savvy web developer can tell you how many kilobytes their code consumes. They bundle, minify, compress and tune the data sent out to within an inch of its life. Wire weight is easy to measure and has a direct impact on your application's launch time. But how many milliseconds does it take the user's computer to parse and load your code once it's arrived? What differences are there between CPUs, operating systems, browsers and plugins? What speed leaks are we overlooking?

The Test
The Libraries
Results
Minification
So What?
Try this at home
Appendix

The Test

As you look at the data below, keep in mind four things:

Your code is not the only code running on the user's computer
Parse-n-load time comes down to available CPU cycles and RAM
The fastest CPUs aren't getting much faster
The average consumer CPU is getting slower

Even if you look at just sexy new hardware it's hard to ignore low-power clients: there are about 50 million netbooks and 43 million iPhones out there, alongside 10-15 million Android devices. Almost all of them are in the 600-1,600 MHz range and have less than 512MB of RAM. A juicy, growing slice of the market wants to use your software on the equivalent of a desktop from 1998. This includes rich Westerners as well as people in the fastest-growing international markets.

The test harness loads a given block of Javascript from a local file over and over and measures the setup time. My test subjects were the Yahoo User Interface (YUI) libraries, Scriptaculous, and jQuery UI. I've also included the main Javascript application code from GitHub.

The core test is as simple as can be: record a start time, load the script, record the end time, and repeat over 1,000 iterations. The tests were run on recently-booted machines with no other programs running. You can check out the project on GitHub and play along at home.

The Libraries

Flickr: charlietakesphotos

YUI 2.8.0r4, 390KB
partial (dom, event, datasource, datatable, layout, tabview, treeview, menu)
YUI 3.0 build 1549, 311KB
"kitchen sink"
Scriptaculous 1.8.3, 159KB
"kitchen sink"
jQuery UI 1.7.2, 359KB
"kitchen sink", minus translations
GitHub.com 09 Feb 2010, 211KB
main application code, including jQuery 1.4

The point of this benchmark is to compare browsers and CPUs. Comparing the parse-n-load of different libraries puts you on shaky ground. For example, the YUI2 libraries are as of this writing more comprehensive than, say, YUI3 or Scriptaculous. On the other hand it's rare for an application to load every module as we're doing here. Also, each library has a diffferent approach to initialization. YUI2 does a lot of work up-front while YUI3 does things more lazily. From there you get into a complex question about what benefits each library buys you. Do not base the choice of library on this benchmark.

MacBook Pro 2.26GHz, OSX 10.5.8

	Chrome	Safari 4	Firefox 3	Firefox 3.5	Firefox 3.6	Opera 10*	Safari 3*
YUI3 (raw)	62	98	114	128	115	112	129
YUI3	31	47	78	94	83	62	111
YUI2 (raw)	23	153	131	142	137	116	150
YUI2	11	110	77	105	102	74	125
Scriptaculous	7	50*	63	91	71
jQuery UI	3	40	84	95	84
GitHub	4	63	67	80	66

Table 0: Parse-and-load times in milliseconds for various Javascript libraries and browsers, 95th percentile mean. MacBook Pro 2.26 GHz Intel Core 2 Duo with 4GB of 1GHz DDR3 RAM.
* Incomplete test, only 100 iterations. See "Tragic Opera" below.

Presario R3000 1.6GHz, WinXP SP3

	Chrome	Firefox 3	Firefox 3.5	Firefox 3.6	IE 6	IE 7	IE 8
YUI3 (raw)	38	163	189	178	118	106	101
YUI3	19	119	143	134	65	61	51
YUI2 (raw)	37	187	220	158	209	132	122
YUI2	18	140	165	148	89	83	69
Scriptaculous	14	89	135	116	82	84	44
jQuery UI	11	119	151	136	62	62	53
GitHub	9	89	110	107	56	54	41

Table 1: Parse-and-load, 95th percentile mean. Compaq Presario R3000 at 1.6GHz, Windows XP SP3 and 512MB RAM.

MacBook Air 1.6GHz, OSX 10.5.8

	Chrome	Safari 4	Firefox 3	Firefox 3.5	Firefox 3.6
YUI3 (raw)	19	92	219	234	216
YUI3	8	72	178	170	157
YUI2 (raw)	24	251	301	268	249
YUI2	15	182	176	190	177
Scriptaculous	10	56	97	163	141
jQuery UI	4	61	150	167	151
GitHub	5	77	131	130	117

Table 2: Parse-and-load, 95th percentile mean. MacBook Air 1.6 GHz Intel Core 2 Duo with 2GB of 1GHz DDR2 RAM.

NB: the MacBook Air had overheating problems so some of these numbers in this table may be skewed upwards. On the other hand, that's precisely the kind of crap your users have to deal with.

There is a noticeable spread between different browsers on the same hardware and OS. Firefox 3.5 is a few points slower than 3.0, but 3.6 improved on that. Internet Explorer is surprisingly fast at parse-n-load across all tested versions. I didn't include standard deviations because aside from some pathological cases they were small. If you run the benchmark for yourself you will get mean average, stddev, and a time series graph for your enjoyment.

Minification FTW

Here is a comparison of the YUI libraries in "raw" form (with comments, whitespace, etc) and the same code minified using YUI Compressor. As expected, minification helps parse-n-load in addition to network transmission time. This is probably due to the absence of comments and extra whitespace.

	Chrome	Firefox 3	Firefox 3.5	Firefox 3.6	IE 6	IE 7	IE 8
YUI3 (raw)	38	163	189	178	118	106	101
YUI3 (minified)	19	119	143	134	65	61	51

YUI2 (raw)	37	187	220	158	209	132	122
YUI2 (minified)	18	140	165	148	89	83	69

Table 3: Code minified with YUI Compressor parses faster.

So What?

If you have a large amount of Javascript in your application it's natural to bundle it all up into one file to save on network transit time and increase cacheability. But if certain parts of your application only use certain parts of the bundle, you might cause the user to unecessarily parse unused code on every page load. The ideal would be to decouple three things that are now tightly wound together: network transit, parse-n-load, and cacheability. There is a lot of work going on in this space but so far there is no silver bullet.

Alexander Limi from Mozilla has a proposal to use zip files for bundling.
The folks at 280North have found a neat way to do multi-file bundling in their Cappuccino framework, using existing technology.
Google is proposing an extension to HTTPS that allows multiple concurrent streams over a single TCP connection.
SproutCore and the Google Mobile Team recently demonstrated ways to load Javascript code as dumb strings that is evaluated at a time of the programmer's choosing.

Try this at home

The Parse-N-Load benchmark is open source and free for use. It's early and I'm sure there are bugs. If you have other kinds of hardware (Netbooks! Windows! Linux!), please try it out and let me know what you find.

Appendix

The Curious Case of Chrome

While Google Chrome appears to be an order of magnitude faster at parse-n-load, the truth may be a little more complex. Running this benchmark in Chrome sometimes produces sharp cliffs in the time series graph, especially on slower CPUs. That might be the V8 engine's inline caching kicking in. I also suspect it could be caching the machine code it compiles on the first pass. Or this could be something silly like the CPU coming out of low-power mode. If anyone who knows more about what's going on can speak up, please do.

Figure 0: Chrome's V8 Javascript engine has two speeds: fast and very fast.

Debugging the Benchmark

The first problem that came up was different blocking behavior between browsers. In Safari 4 (but not 3) if you create a script tag that points to an external file, that action will block, ie, wait until that file is completely parsed and loaded. This makes timing it very easy. In Firefox, however, this action asynchronous: the statement that creates the script element returns immediately and the file is loaded in a separate thread. This means you have to set up a callback in the separate thread to both measure elapsed time and kick off the next iteration of the test. You have to be careful not the blow the stack with too many nested function calls. Google Chrome is also asynchronous and has an altogether different stack behavior. If all that wasn't enough, browsers have very different memory allocation behavior, of which more (oh, much more) below.

Adventures in Garbage Collecting

Every browser seems to have a different system for allocating memory while parsing Javascript code. When you graph the results from Safari 4 this is what I saw initially:

Figure 1: the effects of GC halts

Interesting. All of the source files in the benchmark are local so that's not I/O wait. The regularity of huge spikes suggests that the browser is pausing every so often to free up memory via garbage collection. When you remove those spikes another interesting pattern shows up. Here is the same graph with the top 5% of datapoints removed:

Figure 2: 95th percentile graph of Javascript load times in Safari 4

It appears that the parse-n-load time of a given piece of Javascript in Safari 4 will increase linearly with the amount of garbage. The load time can grow as much as 3X longer than normal before GC kicks in. I'm not certain whether this is an artifact of the benchmark or if it actually happens a lot during real-world use. When I added code to explicitly delete the previous script tag before creating a new one, the sawtooth elongated but did not go away. Other browsers exhibit similar halt-the-world GC behavior but only Safari 4 and Opera 10.5alpha have this sawtooth. Firefox's graph stays fairly horizontal but has many more small spikes:

Figure 3: 95th percentile graph of Javascript load times in Firefox 3.5.3

A Tragic Opera

The results for Opera 10.0 and Safari 3 have asterisks because I couldn't get them to run the complete test. Opera got steadily slower up to 250 iterations after which it started serious thrashing and had to be killed. Opera 10.5alpha reportedly has a new Javascript engine, and some quick tests show more sawtooth-like behavior, but it still is not able to complete the full 1,000 iterations before spinning out of control.

Figure 4: Opera 10 choking on the parse-n-load benchmark

Safari 3 hit the wall even earlier, triggering an Out-of-Memory error after a couple hundred iterations. For some reason I had similar problems with Safari 4 on the Scriptaculous test, which is why that cell has an asterisk.

Figure 5: Safari 3 got to this point, then threw an OOM

Many thanks to Adrien Arculeo, Ryan Tomayko of GitHub, Robert Halliday, Eric Miraglia, Lucas Smith, and random internet heroes for their help, ideas and corrections.

The Soul of a Portable Machine

2009-12-18T10:49:00.000-08:00

Instead of lugging a laptop around, why not carry just the bits? I don't mean keeping your files on a USB key, or even booting from an external drive. I mean storing the entire "soul": the applications, data, OS and RAM image as a virtual machine you carry around on a portable drive. You could plug it into any sufficiently powerful host and resume your work without interruption [0].

So, I tried it. My primary "computer" is a $25 USB flash drive that I plug into $2,500 dumb terminals.

Setting it up

Note to OSX users: Apple does not want people running their operating system under virtualization. An OSX install disc will not normally boot under VMWare or Parallels. This policy is stupid and overbearing but I don't expect Apple to change, any more than I expect the control-freak leader of a hippie commune to loosen his grip on the bag of peyote buttons [1].

I started off with a 4GB stick I had lying around and an Ubuntu Linux image running under VMWare. It was fine for browsing, code editing, etc, but was too space-constrained for real work. The total size of your virtual machine files will be the disk size + RAM size plus a few percent overhead. VMWare requires additional free space equal to your RAM size, presumably in order to write the contents of RAM to disk atomically during a "suspend". So a 2GB virtual machine image with 1GB of RAM allocated will require more than 4GB of space. This turned out to be a nice excuse to buy myself a 16GB stick.

Setting up corporate effluent pipes (email, calendaring, VPN) on the VM would have taken more time than I wanted to spend, so I run them directly on the hosts.

Make sure your desktop background, window themes, terminal colors, etc are different between your host and virtual machines so you don't get confused. Turn off the VM's screensaver.

Use a pre-allocated disk. I made a single-file disk image instead of multiple 2GB chunks, but I have no idea if it makes a difference to speed or error-resistance.

It's like a laptop, except without the laptop

Running my Linux VM directly from a USB drive is smooth, contrary to expectations. I thought I would have to sync my machine image to the host machine's hard drive to get decent performance. I now put my computer in my pocket, commute to the office, plug in, and start working. Over lunch I hibernate the VM and make a backup to the host machine. If I need to work at home I plug it into my personal computer.

Annoyances & Limitations

The VM is slower than running directly on the host machine, but not by much unless you are doing disk-intensive work like compiling MacPorts. Video performance feels sluggish when scrolling or moving windows.

A usable OSX image is too large for my USB drive but a cheap external hard drive works well enough. Next year 64GB USB and SD drives will be in the sweet spot.

My work often requires testing on Windows inside other virtual machines, so I can't use all available RAM for my primary VM. There are minor UI annoyances like the host menubar popping up on OSX. A bug with the virtual sound system in VMWare seems to trigger a long-standing bug in OSX/Flash: YouTube videos play for a second then halt.

Next steps

It would be nice to transform an existing OSX system into a virtual one. I believe you could back up your machine using CarbonCopyCloner or SuperDuper to a .dmg file, then use qemu-img to convert to a .vmdk or other virtual disk format. But unless you find a way to shrink the image you'd need a portable drive with a capacity larger than your current hard drive.

The next big leap would be to replace the USB key with a real portable computer, something like the SmartQ5. The idea is to have one virtual machine that takes full advantage of whatever hardware it happens to be running on. Running a desktop-class virtual machine on a tiny device like would be extremely tricky if not impossible. Even leaving aside the different CPU, it's not possible (as far as I know) to change the amount of RAM or number of CPUs assigned to a VM without rebooting it. But I believe the hacks are out there, just waiting to be discovered.

Notes

[0] See Tom Hughes-Croucher's state vs sync and also the IBM SoulPad project from 2005.

[1] There are many ways to run OSX inside a VM, most of which are against the EULA in jurisdictions where the EULA is enforceable. The easiest way is to install a licensed generic copy of OSX (not the special model-specific copy that comes with Macs) on VMWare or Parallels, and then add some "special touches".

XCode (and thus gcc/make, MacPorts and the rest of the toolchain) will not install on a virtual OSX machine with the "ServerVersion.plist" hack enabled, giving a "error while evaluating JavaScript for the package" error. Software updates also fail. The point of the hack is to misrepresent the type of OSX you are running. So make a pair of scripts to toggle it. Further, deponent sayeth not.

A Dismal Guide to DNS

2009-11-24T09:45:00.000-08:00

(Originally appeared on Yahoo's Developer Blog)

The Domain Name System (DNS) is part of the "dark matter" of the internet. It's hard to observe the DNS directly yet it exerts an obscure, pervasive influence without which everything would fly apart. Because it's so difficult to probe people tend to take it for granted, which I think is a mistake. DNS problems can hurt the speed and reliability of your applications without you even noticing. In this article we'll take a look at the behavior of the DNS and walk through some experiments you can run to gather valuable data about your users' network performance.

A Clever Shambles

Before two computers can talk to each other on the 'net, one of them has to know the numeric IP address of the other. Using the DNS is often compared to looking up a number in the phone book. But that can give the impression the information is in one place, close to hand.

Instead, imagine it's 1982. You live in Tucson and you want to call a hotel in Toronto. You don't have a Toronto phone book so you call your local library. They don't have one either. Life is boring in Tucson, so the librarian uses her New York phone book to call another library. The nice lady in New York looks up the hotel's number in her copy of the Toronto phone book, tells it to your local librarian, who then calls back to give it to you. Doing all this is a hassle, so everyone in the chain writes down the number just in case the question ever comes up again.

The DNS is even more complex because of the hierarchy of internet domains. Consider the host name foo.bar.example.net. To look it up your computer will have to look up every part of the name, in reverse order. That means resolving ".", then "net.", then "example.net.", "bar.example.net.", and finally "foo.bar.example.net."[0]. It's not just a matter of finding the Toronto book. It's looking up someone who knows someone who has the Canada book and from there who has the Ontario book, then the Toronto book, and so on.

If this sounds ridiculously complex and fragile, that's because it is. Writing down the answer to common queries, aka caching, is the only reason we're able to get away with it. In practice the root domain "." is known to everyone. During normal operation "net." should be cached all levels including at your local librarian, aka your ISP. Anything beyond that requires some lookups unless the domain is already very well-known.

How long does it take to look up a hostname?

A single DNS lookup may involve several recursive lookups to machines all over the world. Because of this hassle, information is cached for short periods of time at every level, including on your computer. So "the time it takes to do a DNS lookup" can vary wildly depending on the state of affairs in many different places, and the quality of the network connections between them.

On Mac OSX the dscacheutil command will tell you about your computer's latency and cache hit ratio:


$ dscacheutil -statistics
Overall Statistics:
    Average Call Time     - 0.118626
    Cache Hits            - 236152
    Cache Misses          - 231052
    Total External Calls  - 279350

Statistics by procedure:

             Procedure   Cache Hits   Cache Misses   External Calls
    ------------------   ----------   ------------   --------------
         gethostbyname       161252          39952             6749
         gethostbyaddr           60            151              211
         ...

Listing 0: Mac OSX DNS statistics

These numbers are interesting but fairly useless for our purposes. It combines cached and uncached lookups into one "average". Also, browsers often cache and even precache DNS information, bypassing whatever the operating system is doing. So we can't rely on what the machine tells us. We need to do some experimenting on our own.

First, I ran long series of tests against Yahoo hostnames from the office, my house, and other locations. For 100 seconds I ran as many DNS lookups as I could and timed them. Each lookup was for a wildcard hostname. A wildcard like *.dnstest.example.net means you can make up random new hostnames on the fly, eg x9zzy.dnstest.example.net, that will resolve to a real IP address. This ensures that each test will be a full end-to-end DNS lookup without any caching to skew the numbers [1].

Figure 0: Average DNS latency at various locations

This graph is useful mostly to illustrate that it's possible for users on "broadband" connections to have invisible performance problems related to DNS. But it doesn't tell you which users or how many.

How can we figure out the response time distribution (distribution, not average) for a wide range of users? How can we get a better idea of the role the DNS plays in the performance of web applications? Conditions on the internet change constantly. The tests would have to be large-scale and continuous to mean anything.

Let's scope things down a bit. We don't really care about how quickly users resolve any hostname. We care about how quickly our users resolve our hostnames. So maybe you can get the data you want by observing your users. Unfortunately DNS lookups happen mostly through computers we do not control. Worse, they happen over UDP, which doesn't expose performance data to the callee. The request and response packets are sent without any error correction or acknowledgement. So we can't just look at the usual logs we collect on our servers.

The librarian in New York will never know how long it took the librarian in Tucson to call you back. The hotel staffer in Toronto has no idea how you found their number. That is, unless you tell him. And that's what we'll do: run a special series of tests from the perspective of the caller, ie the users, and report back results.

A DNS Observatory

It's tricky but not impossible to gather some statistics on user DNS latency without running benchmark software on their computers. One way works like this:

Set up a wildcard hostname, perferably one that does not share cookies with your main site. Give it a low TTL, say, 60 seconds, so you don't pollute downstream caches.

Set up a webserver for the wildcard hostname that serves zero-byte files as fast as possible. Make sure that KeepAlive, Nagle, and any caching headers are turned off.

In the footer of the pages in your main site, add a script similar to Listing 1. It performs two HTTP requests: /A.gif and /B.gif. The first image load, A, will require a full DNS lookup and an HTTP transaction. The second, B, should only involve an HTTP transaction.

Subtract the time it takes to complete B from the time it takes to complete A, and you have a (very) rough idea of how long it took to perform just the DNS lookup.

Send the DNS and HTTP statistics back to your server as part of another image request. You can extract the results later from your logs.

Rinse and repeat over a large sample (>10,000) of page views. Millions if you can.

NB: You will get strange, even negative, numbers from this test. The deviation of individual data points can be greater than the phenomenon you are trying to measure. If you want to get accurate numbers for a specific user you'll need to run many tests over a period of time. But a single test per user works well enough in aggregate.


<script>
(function() {
  function dns_test() {
    var random = Math.floor(Math.random()*(2147483647)).toString(36);
    var host = 'http://'+random+".dnstest.example.net";
    var img1 = new Image();
    var img2 = new Image();
    var img3 = new Image();
    var ts = null;
    var stats = {};
    img1.onload = function() {
        stats['dns'] = (new Date()).getTime() - ts;
        ts = (new Date()).getTime();
        img2.src = host + "/B.gif";
    };
    img2.onload = function() {
        stats['http'] = (new Date()).getTime() - ts;
        stats.dns = stats.dns - stats.http; // the clever bit
        img3.src = host + '/dnstest.gif?dns='+stats.dns+'&http='+stats.http;
    };
    ts = (new Date()).getTime();
    img1.src = host + "/A.gif";
  }
  window.setTimeout(dns_test, 11337);
})();
</script>

Listing 1: A poor man's DNS observatory

Below is a graph of the distribution of uncached DNS lookup times from real users in the wild, collected by this script over one week. The sample was heavily skewed towards US broadband connections. The median was 146 milliseconds and the geometric mean was 163 milliseconds [2]. This is rather larger than the 20-120 milliseconds quoted in the Yahoo Performance Guidelines for a "typical" DNS lookup. Beware pithy numbers (even ours).

The distribution is even more interesting than the averages. Twenty percent of users in our sample took more than 500 milliseconds just to resolve one hostname. Granted, these lookups were uncached. Assuming a 50% cache hit rate, that's still one out of ten users in this dataset laboring under crappy DNS performance. As of this writing that's a market as large as Safari, Chrome and Opera combined.

Figure 1: A histogram of uncached DNS request times

The cause is unclear. It's possible that user network quality is just that bad. It could be physical distance. It could also be the DNS resolvers of ISPs at fault. It could be your DNS server. Or it could be something else. Or all of the above.

Remember that your mileage may vary. Not every combination of site and userbase will have a similar graph. Also remember that a lot of caching is going on at every level of the system. There's not a simple fixed cost to using alternate hosts for your images and scripts. The best strategy may well be to have one and only one "asset host" or CDN that does not share cookies with your main site.

If you run a commercial website, consider setting up with a dedicated DNS hosting provider that has presence on several continents. The DNS hosting service typically thrown in for free by domain registrars is not very good. For most sites, solid DNS hosting costs about $USD 50 per year. It's worth the effort. Heck, set up with two different services for failover.

Try this at home

For privacy reasons we can't release the raw data we collected. But if you have a website with a fair amount of traffic, I strongly encourage you to run these DNS measurements for yourself. You can learn a lot by drilling down into the data.

Play with graphing the distributions of different subnets (eg 18.* for MIT or 12.* for AT&T). You might be surprised at who is fast and who is slow.

In your webserver logs for /dnstest.gif there should be a User-Agent field as well. So you can look at correlations between DNS performance, browsers, and operating systems. For example, check out those little bumps at 1s and 3s in Figure 1. It turns out that the DNS resolver in Windows has aggressive timeouts. Those bumps are caused by Windows clients timing out then succeeding on a retry.

We're not just timing DNS latency, we're also timing how long it takes to perform a minimal TCP handshake + HTTP transaction. That gives you interesting information about user connection latencies, for free. But that's a whole 'nother subject.

This article is the second in a series and part of ongoing research on web app performance. If you have any suggestions or ideas to help improve the experiments, please leave a note in the comments. Next we hope to dig into more detail about user network performance data and how you can use it to improve your websites and applications.

Notes

[0] The dot "." at the end is not a typo. Though "com" and "net" are called "top-level" domains there is actually one more layer behind them called the root domain, designated by that trailing dot. The root domain is managed as a global public utility by dozens of internet service providers all over the world.

Fun fact: the entire country of Sweden dropped off the 'net in October 2009 because a network operator forgot to include that last dot in a configuration file.

[1] I'm fudging here a bit. It's possible that during this test, everything up to .dnstest.example.net will be cached at the user's ISP. This is by design, to reduce load on the root and top-level domain servers. But the lookup should always at least do a request to the ISP's resolver and a request in turn to example.net's authoritative DNS server.

[2] These kinds of datasets tend to be log-normal, with long thin tails trailing from a large central spike. The "average" value, or arithmetic mean, would be misleading in this case so we won't discuss it.

[∞] Bonus footnote! Here is the code to generate a table from your webserver logs:


# run a grep for "/dnstest.gif" and save to a file
bzcat /your/apache/logs/access_log.*.txt.bz2 | grep dnstest.gif > /tmp/dnstest.log

# perl magic to split the data into columns
echo "A,B,C,D,dns,http" > latency.csv
perl -lane 'if (/^([\d]+)\.([\d]+)\.([\d]+)\.([\d]+).+dns=(\d+)\&http=(\d+)/) { print "$1,$2,$3,$4,$5,$6" }' dnstest.log >> latency.csv

Here are the R commands to generate graphs and poke around at the data. If you've never tried R, it is a wonderful open-source statistics suite. The best introduction on how to use it is here. (PDF)


## R script for generating the histogram
x <- read.csv("latency.csv", header=TRUE)

# take only results greater than 0 and less than 4,000
y <- subset(x, (x$dns > 0 & x$dns < 4000))

# draw the histogram 
hist(y$dns, xlab="Milliseconds", main=NA, breaks=200, col="red", border="red", prob=FALSE)

# rug() adds "tassles" to the bottom of the graph to show data point density.
# rug() makes tassles. Get it? Yeah.
rug(sample(y$dns, 5000))


## bonus bonus: more interesting stats

# Show only requests from Brazil. this filter is not strictly true
# (ie, they have more subnets than 200/8) but it's true enough to play with.
brazil = subset(y, (y$A==200))
hist(brazil$dns, xlab="Milliseconds", main="Brazil (200.*)", breaks=200, col="green", border="green", prob=FALSE)
rug(sample(brazil$dns, 5000))

## other interesting subnets
mit = subset(y, (y$A==18))
att = subset(y, (y$A==12))

# the "average" is not very useful with log-normal datasets
#mean(y$dns)

# median and geometric mean are more informative
median(y$dns)
exp(mean(log(y$dns)))

# percentage of users over / under a specific point
table(y$dns > 250) / length(y$dns) 
table(y$dns > 500) / length(y$dns)

YCombinator's RFS #5: An Accidental Case Study

2009-11-21T11:21:00.000-08:00

Update: check out ddotdash.

I pricked my ears up at the latest Request For Startups from YCombinator. Last year I wrote the first 6,000 lines of my last startup on an eeePC 901, which is about half the size of a MacBook Air. It's cheap and solid-state so I treated it more like a largish book than a smallish computer. I toted it around to many more places, even parties. Losing that thread of worry about damage made it feel an extra half-kilo lighter. I sometimes wished it had a cell phone built in.

In the end the BabyBook was too cramped for long sessions. Both my code output and my hands suffered. The RFS's accompanying essay is correct that whatever platform hackers use to hack on eventually wins in the larger market. But it doesn't necessarily follow that because hackers have smartphones the next step is hacking directly on them. An equally plausible path is that our phones become portable homedirs that plug into any computer when we need to do real work.

Any human input surface smaller than a sheet of paper starts to bump against physical limitations that Moore's law can't fix. To do development on a mobile you have to be willing to give up both touch-typing and large screens. That's a big sacrifice, and it's hard to say what benefits I'd trade for them.

Any innovation that improves the speed of reading or writing code on a small device should also work on a large device, unless it takes advantage of features or use cases unique to the small one. Otherwise it's not a net advantage to mobile hacking. I giggle at the thought of using the accelerometer to indent, smacking my code into line. But the list of things an iPhone can do that a laptop can't is short while the inverse is very long.

Was the OQO early, or wrong?

Let's start fresh. Imagine you have a device the size of a thick passport or Moleskine. It is a phone plus a computer as powerful as a laptop from three years ago. You can hack on it directly in a pinch. It has enough storage for all of your working projects. You can plug it into any keyboard and screen (or computer) and start hacking.

You know what? That sounds pretty neat. And we'll probably get there soon. Perhaps someone will make a dongle that does this for iPhone or Android. Or it could be the accessory that saves the FreeRunner. But this is not very different from today, except for the reduced utility of your computer when you are not at a properly-equipped desk.

So let's go back to code input and output. We use keyboards because we have 100 years of experience with them as speed-of-thought input devices. We use ever-larger screens in order to view ever-larger amounts of code and data in one glance, partly because moving our eyes is faster than moving our hands. Random access matters a lot.

Multitouch gestures (pinch, zoom, scroll, etc) are interesting and so far underused. Using the "mouse" is less of a productivity hit on a phone because your fingers are already in position. Instead of representing code as long lists we spread out like so much wallpaper, maybe we can represent and navigate it like the directed multigraph it actually is. The Canon Cat and Archy might point the way here.

That leaves speed of input. Hinting and autocomplete might bring the mobile programmer back up to an acceptable wpm. Terser languages and WYSIWYG UI builders can help as well. But still I worry about fatigue. It's hard enough to work comfortably on the hardware we have. On the other hand, there's no reason we need to have a keyboard per se. Experienced telegraph operators could type up to 40 wpm with one finger using Morse code. Typing on virtual keyboards is slow because unlike keyboards or game controllers you have to carefully coordinate vision with motion. Instead, I wonder what one could do with two thumbs, an updated Morse, and a little practice.

YUI Tip: Using DataTable and TreeView together

2009-10-17T14:09:00.000-07:00

For this tip we will make a browser for web server logs. The TreeView will display file and folder paths, and the DataTable will display individual log lines. Clicking on a file or folder in the Tree will cause the DataTable to filter out all but that path.

Read the rest at Yahoo's UI blog.

A Dismal Guide to Bandwidth

2009-10-07T17:41:00.000-07:00

(Originally appeared on Yahoo's Developer Blog)
Web app developers spend most of our time not thinking about how data is actually transmitted through the bowels of the network stack. Abstractions at the application layer let us pretend that networks read and write whole messages as smooth streams of bytes. Generally this is a good thing. But knowing what's going underneath is crucial to performance tuning and application design. The character of our users' internet connections is changing and some of the rules of thumb we rely on may need to be revised.
In reality, the Internet is more like a giant cascading multiplayer game of pachinko. You pour some balls in, they bounce around, lights flash and —usually— they come out in the right order on the other side of the world.

What we talk about, when we talk about bandwidth

It's common to talk about network connections solely in terms of "bandwidth". Users are segmented into the high-bandwidth who get the best experience, and low-bandwidth users in the backwoods. We hope some day everyone will be high-bandwidth and we won't have to worry about it anymore.
That mental shorthand served when users had reasonably consistent wired connections and their computers ran one application at a time. But it's like talking only about the top speed of a car or the MHz of a computer. Latency and asymmetry matter at least as much as the notional bits-per-second and I argue that they are becoming even more important. The quality of the "last mile" of network between users and the backbone is in some ways getting worse as people ditch their copper wires for shared wifi and mobile towers, and clog their uplinks with video chat.
It's a rough world out there, and we need to to a better job of thinking about and testing under realistic network conditions. A better mental model of bandwidth should include:

packets-per-second
packet latency
upstream vs downstream

Packets, not bytes

The quantum of internet transmission is not the bit or the byte, it's the packet. Everything that happens on the 'net happens as discrete pachinko balls of regular sizes. A message of N bytes is chopped into ceil(N / 1460) packets [1] which are then sent willy-nilly. That means there is little to no difference between sending 1 byte or 1,000. It also means that sending 1,461 bytes is twice the work of sending 1,460: two packets have to be sent, received, reassembled, and acknowledged.

    Packet #1 Payload
    
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    .....................................................................
    ...........................................................

    Packet #2 Payload
    .

Listing 0: Byte 1,461 aka The Byte of Doom
Crossing the packet line in HTTP is very easy to do without knowing it. Suppose your application uses a third-party web analytics library which, like most analytics libraries, stores a big hunk of data about the user inside long-lived cookie tied to your domain. Suppose you also stuff a little bit of data into the cookie too. This cookie data is thereafter echoed back to your web server upon each request. The boilerplate HTTP headers (Accept, User-agent, etc) sent by every modern browser take up a few hundred more bytes. Add in the actual URL, Referer header, query parameters... and you're dead. There is also the little-known fact that browsers split certain POST requests into at least two packets regardless of the size of the message.
One packet, more or less, who cares? For one, none of your fancy caching and CDNs can help the client send data upstream. TCP slow-start means that the client will wait for acknowledgement of the first packets before sending the rest. And as we'll see below, that extra packet can make a large difference in the responsiveness of your app when it's compounded by latency and narrow upstream connections.

Packet Latency

Packet latency is the time it takes a packet to wind through the wires and hops between points A and B. It is roughly a function of the physical distance (at 2/3 of the speed of light) plus the time the packet spends queued up inside various network devices along the way. A typical packet sent on the 'net backbone between San Francisco and New York will take about 60 milliseconds. But the latency of a user's last-mile internet connection can vary enormously [2]. Maybe it's a hot day and their router is running slowly. The EDGE mobile network has a best-case latency of 150msec and a real-world average of 500msec. There is a semi-famous rant from 1996 complaining about 100msec latency from substandard telephone modems. If only.

Packet loss

Packet loss manifests as packet latency. The odds are decent that a couple packets that made up the copy of this article you are reading got lost along the way. Maybe they had a collision, maybe they stopped to have a beer and forgot. The sending end then has to notice that a packet has not been acknowledged and re-transmit.
Wireless home networks are becoming the norm and they are unfortunately very susceptible to interference from devices sitting on the 2.4GHz band, like microwaves and baby monitors. They are also notorious for cross-vendor incompatibilities. Another dirty secret is that consumer-grade wifi devices you'll find in cafés and small offices don't do traffic shaping. All it takes is one user watching a video to flood the uplink.

Upstream < Downstream

Internet providers lie. That "6 Megabit" cable internet connection is actually 6mbps down and 1mbps up. The bandwidth reserved for upstream transmission is often 20% or less of the total available. This was an almost defensible thing to do until users started file sharing, VOIPing, video chatting, etc en masse. Even though users still pull more information down than they send up, the asymmetry of their connections means that the upstream is a chokepoint that will probably get worse for a long time.

A Dismal Testing Harness

Figure 0: It's popcorn for dinner tonight, my love. I'm doing science!

We need a way to simulate high latency, variable latency, limited packet rate, and packet loss. In the olden days a good way to test the performance of a system through a bad connection was to configure the switch port to run at half-duplex. Sometimes we even did such testing on purpose. :) Tor is pretty good for simulating a crappy connection but it only works for publicly-accessible sites. Microwave ovens consistently cause packet loss (my parents' old monster kills wifi at 20 paces) but it's a waste of electricity.
The ipfw on Mac and FreeBSD comes in handy for local testing. The command below will approximate an iPhone on the EDGE network with a 350kbit/sec throttle, 5% packet loss rate and 500msecs latency. Use sudo ipfw flush to deactivate the rules when you are done.

    $ sudo ipfw pipe 1 config bw 350kbit/s plr 0.05 delay 500ms
    $ sudo ipfw add pipe 1 dst-port http

Here's another that will randomly drop half of all DNS requests. Have fun with that one.

    $ sudo ipfw pipe 2 config plr 0.5
    $ sudo ipfw add pipe 2 dst-port 53

To measure the effects of latency and packet loss I chose a highly-cached 130KB file from Yahoo's servers. I ran a script to download it as many times as possible in 5 minutes under various ipfw rules [3]. The "baseline" runs were the control with no ipfw restrictions or interference.

Figure 1: The effect of packet latency on download speed

Figure 2: Effect of packet loss on download speed

Just 100 milliseconds of packet latency is enough to cause a smallish file to download in an average of 1500 milliseconds instead of 350 milliseconds. And that's not the worst part: the individual download times ranged from 1,000 to 3,000 milliseconds. Software that's consistently slow can be endured. Software that halts for no obvious reason is maddening.

Figure 3: Extreme volatility of response times during packet loss.

So, latency sucks. Now what?

Yahoo's web performance guidelines are still the most complete resource around, and backed up by real-world data. The key advice is to reduce the number of HTTP requests, reduce the amount of data sent, and to order requests in ways that use the observed behavior of browsers to best effect. However there is a simplification which buckets users into high/low/mobile categories. This doesn't necessarily address poor-quality bandwidth across all classes of user. The user's connection quality is often very bad and getting worse, which changes the calculus of what techniques to employ. In particular we should also take into account that:

Upstream packets are almost always expensive.
Any client can have high or low overall bandwidth.
High latency is not an error condition, it's a fact of life.
TCP connections and DNS lookups are expensive under high latency.
Variable latency is in some ways worse than low bandwidth.

Assuming that a large but unknown percentage of your users labor under adverse network conditions, here are some things you can do:

To keep your user's HTTP requests down to one packet, stay within a budget of about 800 bytes for cookies and URLs. Note that every byte of the URL counts twice: once for the URL and once for the Referer header on subsequent clicks. An interesting technique is to store app state that doesn't need to go to the server in fragment identifiers instead of query string parameters, e.g. /blah#foo=bar instead of /blah?foo=bar. Nothing after the # mark is sent to the server.
If your app sends largish amounts of data upstream (excluding images, which are already compressed), consider implementing client-side compression. It's possible to get 1.5:1 compression with a simple LZW+Base64 function; if you're willing to monkey with ActionScript you could probably do real gzip compression.
YSlow says you should flush() early and put Javascript at the bottom. The reasoning is sound: get the HTML <head> portion out as quickly as possible so the browser can start downloading any referenced stylesheets and images. On the other hand, JS is supposed to go on the bottom because script tags halt parallel downloads. The trouble comes when your page arrives in pieces over a long period of time: the HTML and CSS are mostly there, maybe some images, but the JS is lost in the ether. That means the application may look like it's ready to go but actually isn't — the click handlers and logic and ajax includes haven't arrived yet.

Figure 4: docs is loading slowly... dare I click?
Maybe in addition to the CSS/HTML/Javascript sandwich you could stuff a minimal version of the UI into the first 1-3KB, which gets replaced by the full version. Google Docs presents document contents as quickly as possible but disables the buttons until its sanity checks pass. Yahoo's home page does something similar.
This won't do for heavier applications, or those that don't have a lot of passive text to distract the user with while frantic work happens offstage. Gmail compromises with a loading screen which times out after X seconds. On timeout it asks the user to choose whether to reload or use their lite version.
Have a plan for disaster: what should happen when one of your scripts or styles or data blobs never arrives? Worse, what if the user's cached copy is corrupted? How do you detect it? Do you retry or fail? A quick win might be to add a checksum/eval step to your javascript and stylesheets.
We also recommend that you should make as much CSS and Javascript as possible external and to parallelize HTTP requests. But is it wise to do more DNS lookups and open new TCP connections under very high latency? If each new connection takes a couple seconds to establish, it may be better to inline as much as possible.
The trick is how to decide that an arbitrary user is suffering high latency. For mobile users you can pretty much take high latency as a given [4]. Armed with per-IP statistics on client network latency from bullet #4 above, you can build a lookup table of high-latency subnets and handle requests from those subnets differently. For example if your servers are in Seattle it's a good bet that clients in the 200.0.0.0/8 subnet will be slow. 200.* is for Brasil but the point is that you don't need to know it's for Brasil or iPhone or whatever — you're just acting on observed behavior. Handling individual users from "fast" subnets who happen to have high latency is a taller order. It may be possible to get information from the socket layer about how long it took to establish the initial connection. I don't know the answer yet but there is promising research here and there.
A good technique that seems to go in and out of fashion is KeepAlive. Modern high-end load balancers will try to keep the TCP connection alive between themselves and the client, no matter what, while also honoring whatever KeepAlive behavior the webserver asks for. This saves expensive TCP connection setup and teardown without tying up expensive webserver processes (the reason why some people turn it off). There's no reason why you couldn't do the same with a software load balancer / proxy like Varnish.

This article is the first in a series and part of ongoing research on bandwidth and web app performance. It's still early in our research, but we chose to share what we've found early so you can join us on our journey of discovery. Next, we will dig deeper into some of the open questions we've posed, examine real-world performance in the face of high latency and packet loss, and suggest more techniques on how to make your apps work better in adverse conditions based on the data we collect.

Notes

[1] ceil(N / 1460) is the same algorithm you use to figure out how many trips it takes to carry your laundry down the stairs. (ceil is geekspeak for rounding up.) Say you have 50 pounds of clothes and the basket holds 13 pounds. 50 / 13 = 3 remainder 11, so you need to make 4 trips. The bigger the basket the fewer the trips. So why not use huge packets? On private networks you might see configurations for "Jumbo frames". But in the wild you have to consider the cost of packet loss, typical message sizes, old or incompatible routers, etc.

That specific number (aka Maximum Segment Size) comes from the maximum packet size (aka Maximum Transmission Unit) of 1,500 octets (aka bytes) set in RFC 1191 (aka Ethernet v2), minus the space reserved for the source and destination addresses, flags, etc. IPv6, which has been coming any day now since the Clinton administration, will probably converge on an MSS of 1,220 or 1,440 in the wild. Point being, we're stuck with tiny packets for the rest of our lifetimes.

[2] DNS can also cause latency. We tend to take hostname lookups for granted, but an ISP's DNS resolvers are often unloved. It once took me several years to convince BellSouth's customer service that one of their DNS resolvers was actually off the network. User DNS problems are doubly nasty because we as application developers can't control or even detect them.

[3] The script was single-threaded and used a new TCP connection for each request. A single restriction was used per run, ie X milliseconds latency or Y% packet loss. The wifi was a Linksys WRT54g at a distance of 5 meters, with standard firmware in 802.11g mode and WPA2 encryption. The uplink was a "6mbps" home cable connection about 50 miles and ten network hops away from the nearest Yahoo caching server, during off-peak hours.

[4] The Google mobile team recently put out an interesting fact: "On an iPhone 2.2 device, 200k of JavaScript held within a block comment adds 240ms during page load, whereas 200k of JavaScript that is parsed during page load added 2600 ms."

Image Credit: Pachinko by the_toe_stubber on Flickr.

Install a DNS resolver on your laptop

2009-02-26T19:02:00.000-08:00

Updates:

I have since been lectured sternly (and correctly) that running your own resolver is a rude thing to do. If you want an alternative, try using 4.2.2.1-4.2.2.6 as your DNS servers instead.
a Meraki employee pointed out that they are at the mercy of whatever resolvers the ISPs of their volunteers use. I remain unconvinced as they seem to add an extra layer of indirection. Also, they run their own hardware and software. There is no reason why they couldn't use alternate DNS servers or even run their own.

Free hotspot internet providers (eg Meraki) can have pretty good bandwidth but still feel slow because their DNS resolvers suck and they don't know it. You'll have great response from an SSH session or webmail but clicking a link to a new site will pause or fail.

Even large ISPs get this wrong. I tried for several years to convince BellSouth that one of their DNS resolvers was down:

"No, my internet is not down. The DNS server is down. I can ping. DNS. Dee Enn Ess. Pee Eye Enn Gee. Do you understand I'm trying to tell you about a bad problem with your system? One of your DNS servers is down. It's been down since 2003 but it's still in rotation. Yes, I restarted my router. Yes, my connection is now working but that's not the poi--". Click. Good times.

Solution: install your own damned resolver. I recommend Dr Berstein's excellent dnscache, part of ~~daemontools~~ djbdns (which itself runs under daemontools). Incidentally, this is also a good idea for your servers if you do any crawling, image fetching, etc. You'd be surprised how much it can help.

Excellent installation instructions here:

http://matt.simerson.net/computing/dns/djbdns-macosx.shtml

http://matt.simerson.net/computing/dns/djbdns-freebsd.shtml

Credit to tlack, who taught me this trick back when I was still figuring out bash.

How To Save The Web

2008-08-25T23:29:00.030-07:00

A strange as it may sound, public record does not exist on the internet. Consider this: it would be impossible for, say, the New York Times to change something it printed in 1997 -- there are hardcopies all over the world. But for nytimes.com it's as simple as a mouse click. So the internet we have today is public but it's not really a record. Healthy public record is the foundation of a free and literate society.

I propose that a loosely-connected network of independent archives, running on personal computers, under the care of self-interested individuals, and sharing common data formats, can in time self-assemble into something that fits the bill.

How we got here

For historical and technical reasons, stuff published on the web lives primarily on servers controlled by the original publisher. It is "distributed" at the time it is asked for, one copy at a time, and that copy is usually discarded. This means that if the original publisher goes away the authoritative source goes too. It also means the publisher, be it a company, government agency, small club or individual, gets to choose how the data is accessed now and in the future. This new power has become very tempting [0]. The mainstream is slowly realizing that stuff frequently disappears by accident and by design [1]:

sports.yahoo.com "...a New York computer security expert who found official Chinese documents that list He's age as 14 years and 220 days... The spreadsheets were taken down off the site recently and He's name had been removed..."

www.talkingpointsmemo.com "When we went to the page for the photograph of President Bush and Abramoff, the page in question had disappeared from the site..."

www.nytimes.com ...Said videos were posted, then mysteriously disappeared from the Edwards Web site, with officials muttering something about campaign finance rules...

perezhilton.com "Pinksky's photo disappeared from the hospital's site on Friday, as the scandal story started to get more legs..."

Wouldn't people notice if things went away?

Most often they don't, and if they do notice, so what? Revisionism and neglect is low-risk. "Orphan" works are sometimes rescued by their fans. But it's easy to forget that these cases are the exception. Intelligent, influential people say things like this: "Once the Internet knows something, it never forgets. This material just doesn't disappear from the Internet if it's sufficiently interesting." [2]

There are serious problems with this idea. The Rosetta Stone didn't survive thousands of years in the desert because of some intrinsic cultural value. It survived because it's made of stone. The UNIX crowd learned this lesson a few years ago to their lasting regret: there are no digital copies of the first four versions of UNIX, only some printouts.[3]

Worse is the implication is that if something is not "sufficiently interesting", if it's not part of the story a society wants to tell about itself, it's worthless. The future disagrees about what is and is not important, and why. That's the defining characteristic of the future. No one today cares what the Rosetta Stone actually says [4], yet it is more important to us (as the key to hieroglyphics) than it was to the society that made it.

The current situation

The current situation in archives is much like the web: uncoordinated, conflicting, changing. The most widespread problem is a paradoxical attitude: most people understand that a centralized web would be unsustainable, but few seem to carry that logic over to archiving.

There are a few public archives. Many national libraries have set up consortia to study the problem. There are search engine caches like Google's. There is the remarkable and far-sighted archive.org. All of them are welcome --in this game, the more the merrier-- but I believe they have various flaws. Google's cache exists for Google's purposes, and is not designed for the long term [5]. The LOCKSS project [6] is commendable in sprit and clever in design, but access to it appears to be limited to select universities and libraries.

Archive.org has two handicaps. It's not actually possible for one organization to curate the web. Second, being a non-profit sitting target, they are forced to take down stuff they do save [7] [8]

In April 2004 The public editor of the The New York Times spelled out the paper's position in an article titled "Paper of Record? No Way, No Reason, No Thanks". He was speaking more against the obligation to print government notices than the idea of public record per se. But all through it he assumes that (a) someone else will do it, and (b) this information (not to mention copies of his newspaper) will always somehow be available to future historians. At the same time his newspaper was telling archive.org to remove nytimes.com from the collection [9].

What the Archive should be

Decentralized & Redundant
Centralized is too expensive and too fragile. Redundancy increases the odds of survival.

Long-Term
If it's not long-term there's not much point. Open, stable formats.

Locally curated
There should be at least as many opinions about what should go into the archive as people using it.

Public & Coherent
A cache is useless if no one can get to it. It should also contain only things that are already public.

Verifiable
Whether by digital signatures or by comparing copies, or both, the archive must be resistant to tampering.

Respectful of privacy
A lot can be revealed by someone's reading list, and the need to anonymize may conflict with the needs of tamper-proofing.

Useful for User 0
It has to be useful even if you are the only user, otherwise there is much less incentive to use and contribute to it.

In short, we need something akin to the web itself: something that can grow without limit, yet does not require much centralized organization. It can be pulled into pieces and operate independently and merge back together. Once it reaches a certain size it, or at least the idea, will be impossible to kill.

In a sense the archive already exists, though in a low-energy state. Part of it lives in the browser cache of everyone's computer. These caches are not coherent, organized, searchable, or public. They also have a lot of stuff in there that is better left private. We have to work around that. But it's a start.

Dowser's approach

"The lost cannot be recovered; but let us save what remains: not by vaults and locks which fence them from the public eye and use in consigning them to the waste of time, but by such a multiplication of copies, as shall place them beyond the reach of accident."
Thomas Jefferson, 18 February 1791

Dowser is a program designed to run on a normal personal computer but operate exactly like a website. Its user interface is a fully-functioning website that is hosted by and only accessible to the local machine [10]. This website enables the user to add pages to their local archive, search many search engines at once (aka "metasearch"), search the local archive, add organizational tags, take notes, export and share, etc. It is not intended as a professional tool for research specialists, but as a helpmeet for "power users": journalists, students, writers, etc, who have a demonstrated need for powerful research tools but do not have the time or inclination to train on an academic-grade tool.

When a user adds a URL to the archive, Dowser will attempt to retrieve the content of this URL by itself, without reference to any private authentication or "cookies" held by the user. This helps to ensure that private data stays private. Optionally, Dowser will download linked images, videos and such, and any linked pages up to a user-defined "link depth". As time goes on, Dowser will occasionally "ping" the URLs it knows about to determine if they have changed. If so it will download the new copy and thus build up a change history.

The program will not allow the user to alter any data in his archive, though it is possible to delete anything. Since the formats are open and simple, a determined user can of course alter the archive data with other tools. We can't stop that, and should not try, but will be an interesting obstacle.

Pulling it together

So far we have only a local archive. What User 0 reads is saved and available only to User 0. So how do we share it worldwide without inviting spam or violating privacy or running afoul of the law, etc?

The seven qualities above are somewhat in conflict with each other. Privacy may conflict with the need to verify copies, decentralization conflicts with system performance conflicts with public access, etc. Centralized archives solve it by doing it all themselves and building up a reputation for trustworthiness. LOCKSS, which is a network of caching servers installed in many universities, depends on the mutual trust of those institutions for verification and limits public access to ward of lawsuits by copyright owners.

You and I aren't smart enough to solve the sharing problem. We should start small and listen to the users, to use the much greater imagination of the group. Make tools to allow User 0 to share with User 1, then User 2, etc, but make them orthogonal to the rest. Allow people to extend Dowser and use it in ways we don't expect.

The common feature of all of the current archiving schemes is that they are hard for a mere mortal to set up. In theory anyone can download archive.org's software and use it to build their own archive. In practice it's very tricky, and not worth the effort for most people to learn the ancillary skills like programming, systems administration, etc. You can't join LOCKSS without being a major university.

Archiving today is a luxury good: complex, balky, expensive, like cars before 1908. When something is a luxury and you want more of it, what do you do? You turn it into a commodity. Make the hard parts easy, do what you can for the harder parts, get something workable into the hands of a much larger group. They will take it from there.

Notes

[0] Even for me. This is the second version of this essay. The first version is here:
http://carlos.bueno.org/save-the-web-v1.html

[1] It is, unfortunately, easy to find cases where data was disappeared. In 30 minutes I compiled this list of instances where something that went missing was important enough to be written about.
http://carlos.bueno.org/disappeared.html
[2] "Paris Hilton's genitals have joined the undead - they will live forever, stalking the Internet until the last plug is pulled on the last network router."
http://craphound.com/cambridge_biz_lectures.txt
For a few weeks in 2007, the film industry fought a futile a battle to contain the dissemination of a short string of numbers.
http://en.wikipedia.org/wiki/AACS_encryption_key_controversy
[3] "Although most of the UNIX source code from before 5th Edition has disappeared, we are fortunate that paper copies of all the research editions still exist..."
http://minnie.tuhs.org/Seminars/Saving_Unix
[4] The Rosetta Stone is a political ad, the equivalent of "your tax dinarae at work": "...he caused the canals which supplied water to that fortress to be dammed off, although the previous kings could not have done likewise, and much money was expended on them; he assigned a force of footsoldiers and horsemen to the mouths of those canals, in order to watch over them and to protect them, because of the [rising] of the water, which was great in Year 8..."
http://www.britishmuseum.org/explore/highlights/article_index/r/the_rosetta_stone_translation.aspx
[5] "Google takes a snapshot of each page examined as it crawls the web and caches these as a back-up in case the original page is unavailable... The Cached link will be missing for sites that have not been indexed, as well as for sites whose owners have requested we not cache their content."
http://www.google.com/intl/en/help/features_list.html#cached
[6] The LOCKSS project, "Lots Of Copies Keeps Stuff Safe"
http://www.lockss.org

[7] People forget that curating the web was Yahoo's original failed idea.
As of this writing chillingeffects.org documents more than 90 "takedown" letters sent to archive.org alone.
http://www.chillingeffects.org/dmca512/notice.cgi?NoticeID=1899

[8] "The Internet Archive is not interested in offering access to Web sites or other Internet documents whose authors do not want their materials in the collection."
http://www.archive.org/about/exclude.php
[9] "Paper of Record? No Way, No Reason, No Thanks", Daniel Orkent, public editor, The New York Times, 25 April 2004
http://query.nytimes.com/gst/fullpage.html?res=9D02E1D8123AF936A15757C0A9629C8B63

There are about 3,400 pages for 2005, 1,800 for 2006, 550 for 2007, and only 30 pages for 2008 from nytimes.com in archive.org's public collection.
http://web.archive.org/web/*/www.nytimes.com
I don't mean to pick on the NYT, but in the last few years they have gone to a lot of trouble to delete themselves.
[10] This design is similar to Google's Desktop search application and several others. In addition to having seamless integration with external websites, this design avoids the tricky & tedious work of writing custom "plugins" for many combinations of operating system and web browser.

Network Distance

2008-06-09T14:04:00.009-07:00

When Sydney is closer than Sao Paulo

It's quite possible to make your living from the internet without really considering how it's constructed. I came across this talking with my friend Aaron. He's a bright guy but like many people he assumes the net is done "with satellites or whatever", or never think about it.

The general structure of the internet pretty similar to the global air network. Picture those glowing arcs connecting cities on the back of airline magazines. Then recall the hour you spent in line before boarding, the hour spent driving to the airport, the two hours from the big hub airport to the smaller city you really want to go to, and how relieved you are to be traveling now instead of last week when a storm in Chicago somehow messed up flights to Los Angeles. That's basically how your data feels, too.

The long-haul internet is, in fact, a series of tubes. Very fragile tubes.

The long-haul cables that run the global economy are less than 2 inches in diameter, buried under railroad tracks, highways, or ocean sediment. Within densely-settled areas the network is relatively redundant. But between countries and across oceans and mountains, data flows through an uncomfortably small set of bottlenecks.

Any minor disaster can damage large portions of the 'net. In 2004 the Miami/Sao Paulo traffic was suddenly re-routed through Washington DC, then New York, then across the ocean to Brussels(!), then back to SP. As late as 1998, a train wreck or wildfire in northern Florida could cut off large parts of Latin America for days. This year Suez suffered multiple failures and re-routing flooded the already overloaded Europe-Asia network for several weeks.

Even on a good day you can see the problem. Look at how a packet of data might travel from San Francisco to Hong Kong:


(start)
Folsom St, San Francisco (1 mile)
Pine St, San Francisco (2 miles)
Pine St, San Francisco
Oakland, CA (10 miles)
Sacramento, CA (80 miles)
San Jose, CA (120 miles)
Oakland, CA (40 miles)
San Jose, CA (40 miles)
Hong Kong (7,000 miles)
(finish)

That poor little packet of data rattled all around California, looking for an uncongested cable over the Pacific. For each hop, a decision is made to send it on to some other place that may have better luck. The system works pretty well under stress which is good because stress is there all of the time.

The long-haul internet is a map of trade volume between cities, that lags up to 20 years behind reality.

This is actually true for all forms of high-volume transport, so there is a lot of history to learn from. Infrastructure is insanely expensive and slow to build even though it almost always pays off in the long run. Short hops between financial/military/political/industrial hubs tend to get built up first. Just look at how many ways there are to travel between New York, Boston, and DC, for example.

Or look at area codes. At the time the precursor to the modern phone system was built, dialing a 1 took 1/9th the time of dialing a 9. Silly, but true. So there was a premium on lower numbers. New York's area code was 212, DC 202, Los Angeles 310, Chicago 312. El Paso, Texas? 915. Anchorage, Alaska? 907. Miami was definitely not a hub at the time, but it was important to the Navy and Air Force. Miami's code is 305.

So if several factors of demography, geography, and politics align, there may form a route of sufficient capacity between two points. If not, too bad. It takes years to build up demand, more years to begin the project, and more and more years to finish it. There are bribes, labor riots, sabotage, political chicanery, etc. The first trans-continental rail link in the US was completed in 1869. Twenty years earlier, people had been hijacking ships in Louisiana to sail around South America and crash-land ashore at San Francisco.

The same dramas play out when cables are planned and laid. The connection between Seattle and Tokyo is excellent. Ciudad Mexico and Dallas? Fairly new and really fast. New York to London? World-class. But try to get an email from Barcelona to Bangalore, and often you'll find that it routes through America. Companies are scrambling to build up Europe-Asia links. They've been scrambling since the late 1980's, and it will be some time before they get there.

The long-haul internet is not a magic leprechaun.

Even if there weren't these human problems there are still fundamental ones. Let's imagine a perfectly balanced world-wide network. You have an internet business based in San Francisco. Your people are in SF, your technology suppliers are in SF, most of your customers are in the United States. You have a small but growing customer base in Hong Kong and China. We have a perfect 'net, so there are no silly congestion problems and there is just as much bandwidth across oceans as between cities. Rack space in HK is twice the price as in SF.

So what is the no-brainer place to put your next server farm, hire people to maintain it, set up office space, pay property and business taxes, etc? Correct. Hong Kong.

No matter how good the internet gets it will never be faster than a small fraction of the speed of light.

Recently I was looking at the server logs of a site located in the US. The response times had a very high variance, which indicates a severe bottleneck somewhere. After a lot of poking around, trying to find the problem within the server farm, I had the bright idea to segment the logs by source country. And there it was: the response times were all over the map because the users were all over the map. From the perspective of the 'net, Stockholm and Singapore are just down the block while India and Sao Paulo are past the moon.

Rules of thumb:
- Light takes about 100 milliseconds to travel 10,000 miles and back.

- A network packet on a good route may take 3 to 10 times that long.

- A network packet has not "arrived" until a return acknowledgment has been sent all the way back.

- The longer and more complicated the route from here to there, the higher the chance (often more than 10%!) that a packet will get lost.

- The larger the file you send, the more packets it has to be chopped up into, the more likely one will be lost and have to be re-transmitted. All else being equal the transit time of a file increases more than linearly to its size.

Evolved Code vs Enterprise Code

2008-02-01T17:51:00.001-08:00

Once upon a time, Adrian Thompson accidentally created a magical artifact. He used a genetic algorithm to evolve a FPGA circuit that would distinguish between signals of two frequencies. The winning specimen was pretty unlikely: it worked on that chip only, and only at certain temperatures. It contained circuits that were not connected to the others, but when he turned those off the damned thing stopped working. The neighboring gates depended on that extra bit of inductance, which varied with a slight flaw in another gate... maybe. No one really knows.

Once upon a time, a small group of hackers evolved a piece of software. They focused so much on doing more with less that they themselves became crucial parts of the implementation: Leo took over ops because he stayed late to avoid traffic anyway. These support emails routed to Bianca because she spoke Spanish and Portuguese. Tatiana had a nicer touch with angry customers. Carlos had the pager because he's good at troubleshooting. Tom made the changes to the template parser because Tom's the genius who wrote it. The bug tracking system was a whiteboard. Every little detail seemed to depend on some idiosyncrasy of the people or their environment. It worked -- somehow. Every startup is a freak accident.

This is why I think Jaron Lanier's bomb-throwing about closed- versus open-source is off-target. When you're talking about the potential for "radical creativity" in a program, it doesn't matter how many people are allowed to read the source code, but how many are required to understand it. Having too many junior hackers on your source tree is just as bad as too many customers pestering you about compatibility.

The fundamental difference between the first two systems and the ones that come out of large organizations is that there was no difference between design and implementation. Enterprise code is designed, vetted, reviewed, implemented, deployed, life-cycled, etc. The purpose is to have no irreplaceable parts. On the other hand, the magic chip and the startup were evolved as fast as possible in a way that could never be exactly repeated.

A funny thing: when an enterprizey programmer and an alpha-hacker look at each other's code, the same thought occurs for different reasons: "How do they get away with this crap?"

The documentation is the code, alpha-hackers say. Don't you see that you're writing your programs in PowerPoint? The process gurus grumble darkly about heroics, maintainability, knowledge transfer, bus accidents. The enterprizey people regard design as something that must be as separate as possible from the expression-as-code. How else can it be maintained by anyone? Take away Tom and the thing no longer works. True, but if enterprizey was always better, BigCo wouldn't get their lunch eaten by wittle bitty startups.

This schism is most sharp when a big company eats a little one. Most startup acquisitions play out like a spider dissecting a fly: gruesomely impersonal and tedious. You just can't get over the mental image of that fat spider pick pick picking away at a violated husk. Yahoo's epic rewrite of RTML, its painful digestion of my company's system, and that poor bastard trying to understand his magical chip are all the same phenomenon.

There are bits that work only there & then, bits that may be impossible for an outsider to grok completely, bits that even the same people would do differently now.

It's the Economy, Stupid.

2007-12-03T09:11:00.000-08:00

At least 50.7% of the voters in Venezuela would rather not give their president dictatorial powers and a shot at life in office.

The "socialist paradise" bribe in the constitutional referendum was transparent: a 6-hour day and some worker's benefits in exchange for astonishing new powers for Chavez, including no term limit and an extension of the term from 6 to 7 years. The 7-year thing really bothered me. The only reason I can think for it is that, being a prime number, a 7-year term would put him out of step. At the times most other leaders would be sweating re-election he would have lots of elbow room.

But Chavez was out of step on the economy. The bribes were to be funded by oil exports, which are strongly linked to the dollar. With the dollar going down, there is enormous inflation pressure in oil-producing countries. Chavez' answer to that was price and currency controls. The official rate is 2,200-odd Bolivares to 1 Dollar, with tight limits on how much you can buy or sell. As of last week the black-market rate is over 5,000:1. To wish inflation away he put in price controls on basic foodstuffs, with the unsurprising result that there is less and less stuff on the shelves.

Just as Solidarity in Poland was partly fueled by the price of sausage, the NO campaign got boost from the scarcity of Doña Arepa. You can shut down the TV stations, you can shoot the protesters are arrest the "traitors", but you can't monkey with the price of milk.

Almost Famous

2007-11-01T15:09:00.000-07:00

Check out this cutie: Linda Goodwin-Nichols

Ms. Nichols is the Mayor of Kissimee, Florida, a real estate agent and a local philanthropist. She is what we at Spock call a semi-famous person. Her data was automatically picked out from thousands of "Linda Nichols" pages and assembled from over 25 separate sources on the web. Pictures, co-workers, political contributions, fax number, mailing address, press articles, etc etc and so forth. A trained researcher at a newspaper would take hours to gather this much information.

Forget about trawling social networks and Wikipedia. Tagging your friends is fun but not very useful outside of your own circle. Famous people are easy because there is so much unambiguous information about them. Regular joes like you and I are easy because there is so little. The real challenge and value of "people search" is getting this level of depth for the several million people who are somewhere in between.

Persai: failed in the cradle

2007-09-02T23:41:00.000-07:00

Persai is Ted D's new startup. Details are still sketchy, and will remain so until these e-tards finally wither away. The basic idea is that Persai will suck on everyone's RSS feed pipe, swish it around and then spit it out. The killer feature is "recommendations", probably based on a classifying algorithm they cribbed from an old IR paper. So not only will it suck for user #1, it will suck for every other user until their personal filter is trained.

A shitty idea deserves a shitty implementation. The whole thing depends on Rails plus a wobbly collection of Java search-engine-y stuff that keeps changing its name and is nowhere near as efficient or reliable as the Google software it tries to imitate. Bonus: Persai will be hosted on Amazon's EC2 for maximum falability. Read my lips, guys: $73/month plus bandwidth for a lamed webserver with no N-O S-L-A. This is what happens when math nerds-turned-Google interns design a server farm.

Congratulations! Your company is pursuing a

(X) pseudo-technical ( ) web 2.0 ( ) ad-based ( ) acquisition-based

approach to making money. Your idea fails. Here is why it fails. (One or more of the following may apply to your particular idea, and it may have other flaws which we couldn't be arsed to point out.)

(X) 80% of blog pages are robot-generated rips and spam
(X) of the remaining 20%, 95% are metoo posts
(X) "Authority" (i.e. "groupthink") is easily gamed and useless in any event
(X) storage and bandwidth are cheap, but not that cheap
( ) moving data to the CPU is still not cheap
(X) No one will pay to use this
( ) Google will not buy you
(X) Nicely-organized shit is still shit
( ) A dating site that actually works means no repeat customers, dummy.

Specifically, your plan fails to account for:

( ) Lack of central controlling authority for blogs
(X) Asshats
( ) Fakesters
(X) Providing actual value
(X) DMCA takedown notices
(X) Criminal investigations & liability
( ) Thundering Herds (not that you have any users)
(X) Scalability
( ) Hyperventilation about paedophiles/kiddie porn/bomb instructions
(X) Undocumented, alpha-quality, Open Source knock-offs of Google infrastructure
(X) Continued availability of Amazon compute services
( ) Languages other than English (you provincial e-tard)
(X) Extremely low marginal cost of blogspam
(X) Clickfraud
( ) Inherent failure of geocoding IP addresses
( ) Armies of worm-riddled, broadband-connected computers
(X) Eternal arms race involved in all filtering approaches
(X) Unpopularity of weird new search engines
(X) User Fatigue And Internet Login Syndrome (UFAILS)
( ) No one wants to be on the same social network as their mom
( ) Extreme stupidity of the internet public
( ) Extreme cupidity of the internet public
(X) Badly-coded blog readers
(X) Badly-coded feeds
(X) False positives that result in the replication of spammy data at your expense. (Thanks, asshole!)
(X) Search Engine payola
(X) Facebook

and the following philosophical objections may also apply:

(X) Ideas similar to yours are easy to come up with, yet all have failed
( ) Blacklists suck
( ) Whitelists suck
(X) PageRank-style ideas are patented, and also suck
(X) Clustering algorithms run in k^2 time or worse, and, furthermore, suck
( ) Other people should be able to talk about Ron Paul without us having to hear it
( ) Profitability should not involve AdSense
( ) Profitability should not involve banner ads or affiliate programs
( ) Userbase growth should not involve spamming address books
( ) Searching should be free
(X) Why should we have to trust you and your servers?
(X) Why should I listen to your sucky algorithm instead of my friends?

Furthermore, this is what I think about you:

( ) Sorry dude, but I don’t think it would work.
( ) This is a stupid idea, and you’re stupid for suggesting it.
(X) Nice try, assh0le! Go back to wanking on 4chan.

Think Twitter is frivolous? Ask the protesters in Venezuela.

2007-06-01T09:11:00.000-07:00

Twitting your bathroom breaks is frivolous. Like the user who used reddit to report on last year's coup in Thailand, the kids on the streets of Caracas who are protesting Hugo Chavez have a real use for it. They are using Twitter, SMS, camera phones, etc to coordinate, keep people safe, and get the word out as things happen on the scene.

Periodismo De Paz (Spanish)

Update: Sis reminded me about the Usenet posts from "kremvax" during the Russian coup in 1991.

Helprin Wants Eternal Copyright

2007-05-22T08:54:00.000-07:00

I have to respond to this. Mark Helprin, an author and occasional pundit, has really jumped the shark in his New York Times Op-Ed about copyrights (cached). He is seriously (though not too intelligently) arguing for eternal copyright protection.

Let's skip over the emotional appeals in the essay, the strange images of a copyleft junta that steals yachts from the mouths of children yet unborn. Helprin has made two errors in his thinking. One is that real property can be equated with other kinds. The other is that ideas can be easily separated from works.

The term "intellectual property" has no legal basis. There is only real property, trademarks, patents, and copyrights. Trademarks are already treated almost the same as real property (with good reason), and he does not seem to be too interested in patents, so let's stick to copyright.

A story is demonstrably different from an office building: I can copy a story with no material loss to the original copy. It is impossible to "own" or "steal" a copyrighted work, in the sense of denying others the use of it by possession. Thus most real property laws simply do not apply, and that is why they are treated differently under the law. A story cannot be contained, moved, hidden, or destroyed. Pretending otherwise is as silly as charging Xerox or Sony with grand theft. (Which, by the way, was actually tried by numbskulls such as the late Jack Valenti.)

Helprin claims that this is unfair. Ford can keep his car factory for as long as it stands. Ay, but there's the rub. Ford owns real things that must be staffed and amortized and upgraded for them to produce more value. A story has no such pesky problems -- it is indeed forever. By his own logic, wouldn't eternal ownership of maintenance-free, non-material, value-producing things be unfair to owners of material ones?

On to the second assertion: "ideas are immaterial to the question of copyright." Pardon? An entire phylum of lawyers lives off of derivative-work litigation. The separation of "idea" from "work" is a matter of judgement (ie expensive trials), not law. I invite Helprin to invest his hard-earned cash in a cartoon featuring a happy-go-lucky mouse in red shorts. I think it'll be a winner.

His conclusion that "no good case can exist for treating with special disfavor the work of the spirit and the mind" is insupportable and false-to-fact. Copyrights and patents already have extraordinary monopoly protection --enjoyed by no other kind of work-- as compensation for their immateriality. Simply by breathing an author gets exclusive right to exploit the industry called copyright, backed up by the full might of every Berne Convention government. To balance this enormous grant of power against the needs of our shared culture, the culture that enriches all of our lives, it is limited in time.

Now we have some deathless, rapacious corporations that would like to have this protection forever, to hold a veto on our culture unto the seventh generation. These same corporations take the lion's share of profit from authors, and then enlist them in their cause! The arrogance of Ozymandias was a rabbit fart in comparison.

The Financial Times weighs in savagely.

Lawrence Lessig's copyleft hordes do too.

Software for the World

2007-04-06T10:05:00.001-07:00

Q: How many Californians does it take to change a light bulb?
A: Just one. He holds it up and the world rotates around him.

In the US border states there used to be a phenomenon called "sombrero politics". During election season the gringo politicians would go to the Mexican parts of town, put on a gaudy sombrero, eat a burrito and give a short speech (in English). Then they would leave until the next election season. That's about where we are now with software. We take a bit of time out of our busy lives, code some functions, test the Unicode support, and go back to the important work.

When a typical US programmer thinks of "internationalizing" software, he does it superficially. Porting your software to another culture is much more complex than porting to another OS or browser. Like the plates of the Earth's crust, language and mythology sit unseen underneath, shaping the mental landscape.

Language indicates profound differences in how people think. For example, there is no satisfying equivalent for the Spanish word "travieso". It denotes a young, puckish, clever, likable little bastard. The trickster archetype is an important part of that culture; in English there is only a fuzzy cloud of adjectives ^[1]. There are emotional states that would take a dozen words to translate. Time is expressed as a quantity; the Anglophone time-as-distance is slightly weird. Compass directions (i.e., "turn North") are deeply weird and complex. The 12-hour clock and mm/dd/yy format are as archaic as feet and pounds. Why are they the default in so much software? Cultural bias in the programmers.

Bias seeps in everywhere. I grew up in the desert. I don't think much about seafood. If I were creating a recipe site I would lump all seafood into one category and put beef, lamb, etc into their own categories. To me, fish is fish. To someone from the coast that's backwards: meat is meat. Fish is one type, shellfish another. There are mussels, lake fish, river fish, seawater and briny. That is a cultural disconnect at work. If only desert-people wrote software, their view would appear to dominate even though there are many more people who live on coasts.

The LikeBetter guys have a test that compares you against all the YCombinator startup founder's scores. From a series of pictures, it inferred a lot of things about me: I am a female who doesn't like spicy food, is uninterested in politics and who doesn't like pranks. Wrong, wrong, wrong and wrong. What I really don't like is snow, and 7 or 8 of the pictures featured brain-damaged people up to their hips in that stuff. Without feedback likebetter is just shouting in an echo chamber.

They got my arrogance right, at least.

Add all these differences up and you get a kind of cultural accent. As with verbal accents, the thicker it is the less people take you seriously. You might still succeed internationally but only by chance or lack of choice. The most popular chat software in South America and much of Europe is MSN. Do you know why? MSN doesn't either. Google's social network, Orkut, was flooded by Brasilians in 2004, even though the site was in English. Even the Brasilians don't agree on why. But consider that Orkut was invite-only, that three big signs of cultural status in Brasil are sexuality, popularity, and connections with the United States... and there are a lot of brasileiros working for Google ^[2].

US programmers often reply that it's the features that matter, not the funny squiggles over the vowels. Business folk will come back with the size of the US market. But that's short-term thinking. Treating entire continents as rounding errors only works when you are the only game in town.

^[1] A reader suggested "rascal", which captures the flavor very well. Try to remember the last time you've heard that word in casual speech.

^[2] Do you know who is big in Japan? Pat Metheney, Frank Zappa, and all the outré jazz composers you would only know if you are a hardcore music geek. In this case it's not US cachet. Experimental jazz has never been widely popular in the States. The conventional wisdom in music schools is that average ear doesn't get it, that Anglophones focus on lyrics over melody. But the average Asian ear is trained from birth on tonal languages in which a slight inflection changes meaning.

Also: Globalization Test

Globalization Test

2007-04-02T07:00:00.000-07:00

If you write software assuming any of the following:

English
Acronyms
Fixed timezone offsets (pssst... it's summer in Brasil and it's tomorrow in Australia)
"AM" and "PM"
Knowledge of US geography (Quick -- name all of the Canadian "states")
Latin-1 or ASCII-7 (We'll just add the funny things over the vowels later)
US-centric telephone & address formats (You mean it's not called a Zip code?)
US-centric methods of payment
US-centric legal frameworks (You can drink at 16 in England?)
Feet, Pounds, Fahrenheit (How's that Mars probe doing?)
Commas for decimals and periods for thousands-markers
"Soccer"
"Foreign" or "International" anything

...report to my cube immediately for re-education with this baseball bat.

Why Popular Sites Are "Ugly"

2007-03-22T15:25:00.000-07:00

It seems as though popular websites, books, songs, movies, etc offend a lot of principles of quality and taste. What's more, it seems to be by design. It is. The funny thing is that this is exactly as it should be. It can't be any other way.

The reason is that people are more different than you usually like to think. The form of a message has to be tuned to the intended effect, to the semantic receptors of the audience. When your audience is large and the common thread between them so thin, the form of a successful message becomes self-parody, an appeal to the basest desires. Nothing escapes this rule. Just look at the highest of highbrow-but-popular stuff (say, Shakespeare or Cervantes) and count the profound ideas shoved next to diarrhea jokes.

I spent a few years designing ads for the phone book and direct mail. It's a no-nonesense business. Personal aesthetic takes a backseat to performance. In a way it's great: you have a single test of effectiveness (i.e., the phone rings or does not ring), reinforced by fast iteration and feedback. Direct Marketing is a real-world example of design by genetic algorithm.

This kind of design does not always produce ugly results. It seems that way because, by definition, you see more wide-audience-targeted material than other kinds. Yachting and Spa ads are pretty posh but odds are good you're not on the list.

Nor is popularity a good measure of quality. The books Tom Sawyer and Huckleberry Finn have the same characters, same kinds of silly scenes, pirates, farce. Yet Sawyer is a kid's book while Finn is a masterpiece. The difference is the stuff Mark Twain wove in between the crowd-pleasers.

I sympathize with esthetes. I am am one. But if you are trying to make a popular website, remember that you are a commercial artist. Commercial art is the most demanding because of its constraints: the worst being that you do not have final say. The audience does. Would Monty Python be as popular as it is without the flying sheep or the road-testing they did on each sketch?

Also: An essay on Whitespace by Mark Boulton

Weird things you must believe in to be an applied mathematician:

2007-03-22T06:41:00.000-07:00

1) Any theorem that is disproved by contradiction must be thrown out.

2) ...except for Bivalence and LEM even though Russel & Gödel kicked the bottom out of them three generations ago. It's just too scary.

3) Probability, that fluffy blue blankie we use where bivalence breaks down, is axiomatic even though it produces no significant results.

4) A logical tautology cannot also be a factual truth.

5) ...but the Principle of Induction always holds, except when it doesn't, and when it doesn't it means there is something new to learn. Why? Because that's the way it has always happened in the past.

Bueno's Law Of Geocoding:

2007-03-19T12:48:00.000-07:00

The accuracy of a given company's mapping application is 100% for the company headquarters, greater than 75% for the homes and hometowns of company employees, and undefined everywhere else.

Startup School & Startup Culture

2007-03-09T07:39:00.000-08:00

Well, I've made it into Startup School this year. I'm genuinely excited, not just for the speakers but to meet the other 649 hackers.

I'm curious to meet Paul Graham in particular. Over email he's always gracious & thoughtful. But --and this is a little seed of a thought that's been growing for a couple of years-- I wonder about the effect he's having on the industry: whether YCombinator is doing good for the art of software as it is for the business.

Take music, the other creative pursuit linked with garages. People learn it on their own or at expensive schools. Groups condense out of the social soup of young people, and it tends to clump in certain cities. They have goofy names. They do it because they love it. The best are able to work magic. A lot of them fail, some make a decent living, a very few become rock stars.

That's the myth. But that's not what actually happens. In the background are competing factions of promoters, managers, coaches, ghost writers, hired experts, financial backers. Without the cooperation of this network your chances of making it are thin enough to cut your wrist. Without them the Beatles would have just been a really good bar band. Generations of jazz and blues geniuses died in poverty because their looks were not marketable. Talent is not closely correlated with success.

So ask yourself: Who makes money in this regime? The backers, the promoters, the hired guns, the music schools, and a pinch of the talents. How does this regime further the art of music? Do you have a radio? Have you used it lately? The internet is a great end-run around the music cartels, but your startup is already on the internet.

YC is backed by serious & smart people. The pitch is soft, the benefits are compelling. They didn't create this situation. They are doing a lot of good: directly to the kids coming to Boston and as inspiration to everyone else. But if you love the art of software, keep your eyes open. Don't assume that what's best for your success is best for your art, or what's best for them is best for you.

Our Discontented Bones

2007-02-26T19:23:00.000-08:00

Today the Miami sun is a hungover showgirl, leaving pink and purple smears on her pillow-clouds. She washes with lazy rain. It drips on Mediterranean collonades and quirky Arabian facades: memories ancient Persia by way of Spain. It drizzles into your Calle Ocho cafe con leche. It puddles the driveways of a hundred thousand SUVs and a dozen kinds of palm trees -- all of them, like most of us, recent imports.

The lure of Miami is how it embraces the international and mocks the cosmopolitan. Another block is another dimesion: Little Haiti, Little Havana, pockets of Europe and Argentina and Hong Kong, people trading driving, praying, building, teeming in and out of one of the Grand Caravanserais of the New World.

On occasion some 4th-generation cracker will read too much into my complexion. He will confide the unmoored feeling that comes every time he sees a billboard written in another language. I will drop into an accent that is comfortable to my tounge yet alien to my ears. It's an unconcious habit. I will share with him in the Brotherhood of Expatriates: a club with no rites or roster but which calls itself into session wherever beleaguered men find the familiar in a strange place. We will sit and sip and I will listen to how is it's a shame to have to taste the Brotherhood while on our native soil.

The first time it happened I threw the ignorant prick out of my house. But that was during the golden time between callow youth and heavy drinking, before I realized that the unmoored feeling was with me always.

People of recently mixed heritage have no homeland in this sense: there is no place, however distant in space and time, peopled with those who share every one of your values. Is the answer a permanent Wanderjahr?

Quoth Nathaniel Hawthorne, a man who got paid by the comma:

"...the years, after all, have a kind of emptiness, when we spend too many of them on a foreign shore. We defer the reality of life, in such cases, until a future moment, when we shall again breathe our native air; but, by and by, there are no future moments; or, if we do return, we find that the native air has lost its invigorating quality, and that life has shifted its reality to the spot where we have deemed ourselves only temporary residents. Thus, between two countries, we have none at all, or only that little space of either, in which we finally lay down our discontented bones."

I didn't come to love this place until I had left, seen what was around, and came back. I needed distance to appreciate this mad soup that seems to be self-organizing into a cultural capital.