A story about computer science
and other improbable things.

“I once made the mistake of generalizing from a single data point. I'll never do that again!”
– Achilles the Logician

Do something very specific with the money.

We raised money to pay for translations of our book. We didn't need money for the English edition. Between my wife and I we had the technical, graphic, and writing skills to produce a book. While it was sometimes hard to work nights & weekends, it was doable.

But we believe that internationalization is as important for books as it is for software. A good translation requires serious amounts of specialized work, so that's what we asked backers to help pay for.

Keep the rewards simple to understand.

Fine-grained price discrimination is a big-league tool. You are not big-league. I suspect that too many different reward tiers causes potential backers to be overwhelmed by choice. With Lauren Ipsum we offered digital copies, signed paper copies and prints of the illustrations. $5 extra for international postage. That was it. Fulfillment was still an undertaking, but more manageable because of the simplicity.

Charge a lot.

Make sure there is a decent profit on every reward tier. Remember taxes, transaction fees, and shipping. My rule of thumb was to charge double the intended retail price for each item. Selling a hardcover for $60 might feel weird at first, but it's perfectly fine. You'll see less of that than you think. :)

The mix of backers for Ipsum was interesting: about 40% bought digital versions, 35% paperback, and the rest hardcovers and prints. The average revenue per backer was $32, and the average fulfillment cost was about $12. Even without the pure-margin digital tier and high-margin print tier we would have made a good sum. This is key.

Avoid timesinks.

Sure, it's fun and whimsical to offer hand-painted t-shirts or home-baked cookies as a reward, but how many of those can you do? The point of Kickstarter is to trade money for time in the other direction. Backers give you money which you give to other people (eg, your landlord), in exchange for time to produce your actual project. Offering labor-intensive rewards that do not move your project forward are a net loss. A good bad example would be “$5 gets you a sticker of your choice”. At first this sounds reasonable. You want to offer something for a small token payment. But even leaving aside the shipping costs, having to match the backer's choice with the sticker with the envelope will eat up all your time.

Exclusive access is the most precious reward.

Consider offering backers exclusive access to whatever you're producing for a month. This is a unique selling point that scales, and costs you absolutely nothing. It's also a good way to get that crucial pre-launch feedback. Think of it as the dress rehearsal.

Ask backers to help you bridge the gap.

The gap between attracting backers and hawking your wares in the general marketplace is huge. You lose almost all of the social credit you built up inside the Kickstarter nest. No one knows you on Amazon, and no one cares how many people liked you on some other site. This is bad, but manageable.

Once you're mailed out the last package, schedule a period of rest before the real work of launching begins. This is where leaving a few weeks of “beta” time between the end of the Kickstarter and your official launch helps. This gives backers time to receive the work, give you feedback, brag about it online, and so on.

Then you open the curtains and put your work on general sale. Send an update to your backers, asking them to rate and review on various sites. They want to help you. They just got their reward and they will welcome the chance to help your project some more.

Kickstarter gives you awesome analytics.

Believe me, you'll miss it later. The most interesting fact was the hit-based nature of backing. There were some sales from people browsing the site itself, but over 90% came from external clicks directly to the project page. If I was able to get an interesting article out in front of a few thousand people, there would be a burst of purchases that decayed in 2-3 days. Ipsum never reached a self-sustaining critical mass to shoot it to $100K or $1M, but we ended up with a respectable $10K.

This data informed the marketing plan post-Kickstarter. None of the marketing plays we tried worked except press hits, and the timing of those hits is as important as their magnitude.

I wish I had product shots like that.

A site called Geefts ("Gifts for web geeks!") has jumped into the game, pretending to have paperback copies of Lauren Ipsum for only $10. Remember, this is a fairly new print-on-demand book, so where would they get stock?

Geefts is also busy spamming Twitter with identical posts from attractive female accounts linking to their "product page". The site is decently designed, even going to the trouble of photoshopping the cover of my book onto some other book.

It's all a lie. When you click their "Geeft it for $10" button it just dumps you to Amazon's product page for the book with a completely different price. But having passed along, of course, their affiliate tag so they get a commission for whatever you happen to buy there.

I understand marketing your own book, and competing with other people marketing their books. But I'm wasting my time battling bots and spammers that pretend to have copies of my own book. They are playing a numbers game, not caring what happens as long a they get their affiliate fee or shipping & handling charge.

All of this is frankly confusing. On one hand it's nice that people are paying attention, and are least hearing about the idea that everyone can and should learn how to program computers. On the other hand, potential readers have to pass through this thicket of deception.

A story about computer science
and other improbable things.

It gets better. There are whole species of other bots that infest the Amazon Marketplace, pretending to have used copies of books, fighting epic price wars no one ever sees. So with “Turing Test” we have a delightful futuristic absurdity: a computer program, pretending to be human, hawking a book about computers pretending to be human, while other computer programs pretend to have used copies of it. A book that was never actually written, much less printed and read.

The internet has everything.

This would just be an interesting anecdote, except that bot activity also seems to affect books that, you know, actually exist. Last year I published my children's book about computer science, Lauren Ipsum. I set a price of $14.95 for the paperback edition and sales have been pretty good. Then last week I noticed a marketplace bot offering to sell it for $55.63. “Silly bots”, I thought to myself, “must be a bug”. After all, it's print-on-demand, so where would you get a new copy to sell?

Then it occured to me that all they have to do is buy a copy from Amazon, if anyone is ever foolish enough to buy from them, and reap a profit. Lazy evaluation, made flesh. Clever bots!

Then another bot piled on, and then one based in the UK. They started competing with each other on price. Pretty soon they were offering my book below the retail price, and trying to make up the difference on “shipping and handling". I was getting a bit worried.

The punchline is that Amazon itself is a bot that does price-matching. Soon after the marketplace bot's race to the bottom, it decided to put my book on sale! 28% off. I can't wait to find out what that does to my margin. (Update: nothing, it turns out. Amazon is eating the entire discount. This is a pleasant surprise.)

My reaction to this algorithmic whipsawing has settled down to a kind of helpless bemusement. I mean, the plot of my book is about how understanding computers is the first step to taking control of your life in the 21st century. Now I don't know what to believe.

It's possible that the optimal price of Lauren Ipsum is, in fact, ten dollars and seventy-six cents and I should just relax and trust the tattooed hipster who wrote Amazon's pricing algorithm. After all, I no longer have a choice. The price is now determined by the complex interaction of several independent computer programs, most of which don't actually have a copy to sell.

But I can't help but think about that old gambler's proverb: “If you can't spot the sucker, it's you.”

Update, 29 Feb: The Goodwill Industries of Greater Nebraska has jumped into the ring, offering a used copy of Ipsum for $12.18. I believe this one. It mentions that their copy is signed by both me and the illustrator. We sold about 250 signed copies during our Kickstarter campaign, and 2 or 3 went to Nebraska.

Translations: Hebrew, Japanese

Like what you read? Lauren Ipsum is a children's story about computer science. Buy a copy, and you'll be helping us translate it into Spanish, Portuguese, and other languages.

When they were safely inside the town walls, the little lizard popped his head out of Laurie’s pocket.
A story about computer science
and other improbable things.

“See what I mean? Let’s hope they don’t figure out what you did to get in here,” Xor said. “So, why are we here?”

“We’re looking for information. Maybe we can find a map or something.”

“Oh,” said Xor. “I was hoping you were going to say food. Why don’t we try this place?”

In front of them was a storefront with a very fancy sign painted on the window:

“Al-go-rith-ms. That sounds like a kind of fruit.”

“Are you always hungry, Xor?”

“Time flies like an arrow, and fruit flies like a banana. Let’s see if there’s a fruit fly problem I can help them solve.”

A bell jingled as she opened the door. “Hello, hello!” the shopkeeper said. “And welcome to my shop. I’m Tinker, and you are looking for a finely crafted algorithm, am I right?”

Laurie looked at the items listed on the chalkboard, but they didn’t make any sense.

“I’m not sure. What is an algorithm? Can you eat it?”

“What? No, it’s just a fancy way of saying ‘how to do something’. But ‘Algorithm’ looks more impressive on the sign,” said Tinker.

Xor turned orange with disappointment.

“How to do something.” repeated Laurie. “In that case, I want to find a sensible way to visit every town.”

“That sounds like an interesting problem. What have you been doing so far?”

Laurie told him about her adventure in the Red-Black Forest and her visit with Eponymous Bach.

“A Hamiltonian path, eh?” said Tinker. “That’s a tough one. I hate to say it, because he sounds like a nice person, but the Wandering Salesman might take a long, long time to finish his tour of all the towns.”

“Oh, no! But why?”

“If you always go to the nearest town you haven’t visited yet, you might miss a town that’s just a little farther away. Then you go to another town that’s closer to you but still farther from the one you missed, and so on. You can end up criss-crossing the whole country to get to the last few towns.”

“That sounds exhausting,” said Laurie. The Wandering Salesman wasn’t so sensible after all! “So how do I find the shortest path?”

“I’ll have a look at what I have in stock. But it might be expensive.”

“I don’t have much money with me,” Laurie said. She took a few quarters from her bag and showed them to Tinker.

He looked at them with surprise. “‘Quarter Dollar.’ Is this money where you come from?”

“Of course it’s money! That’s seventy-five cents.” she said.

“Cents? We use Fair Coins here.”

“What’s a Fair Coin?”

“Well, they are a bit bigger than these ‘Quarter Dollars’ of yours, but they are not nearly as pretty! You can tell genuine Fair Coins because they always flip heads or tails, fifty-fifty.”

“But you can flip quarters fifty-fifty too!”

“That may be true, but I can’t just take your word for it, can I? Here, all Fair Coins must be certified Fair.”

Laurie was crestfallen.

“Don’t look so sad! I do want to help you,” said Tinker. “Maybe we can do a trade. It so happens I’m in the market for a particular algorithm.”

“But I don’t have any algorithms, either.” said Laurie.

“That’s not a problem,” said Tinker. “You can compose new ones any time you want, with a little bit of thinking.”

“I can? How?”

“Well, everyone develops their own style. You can put little ideas together to make big ideas. Or you put two ideas side-by-side and compare them. Or you start with big ideas and take them apart.”

“You mean like Eponymous does?”

“Yes, just like her. She’s a great Composer.”

Laurie had never thought that she could do things like that herself. But Tinker seemed to think it was normal.

“So what do I do?”

“The algorithm I’m looking for is how to draw a circle, Tinker said. “It’s a tough one, so you’ll have to use your imagination. I’ve asked all the adults and even Ponens and Tollens already, but all they do is mutter about X squared plus Y squared and never get anywhere.”

“Take a look at this.” He handed her a windup toy animal. It had a Shell, and was Round and Green. “This turtle can do three things: it can move forward or backward, it can turn, and it can draw a little dot on the paper.”

“Hey, that’s pretty neat!”

“Yes, but the thing is, it doesn’t know how to do anything else. That’s where the algorithm comes in.” Tinker took out a piece of paper and wrote what looked like a little poem:

Go forward one inch,
make a mark,
repeat five times.

Then he wound up the turtle and placed it on the poem. It went zzzrbt bzzaap whuzzzsh, and so on. Then it drew a line of dots, just like the poem said:

“You see?” Tinker said. “If you put little ideas together, you can make bigger ones,” Tinker said. “And you can compose those ideas into even bigger and bigger ones.”

“How do you do that?” asked Laurie.

“By giving them a name. You can use the name like a handle. Here, let’s call the first idea LINE. Then you can put four lines together to make a square:”

LINE:
Go forward one inch,
make a mark,
repeat five times.

SQUARE:
Make a LINE,
make a right turn,
repeat four times.

Make a SQUARE.

The little turtle zzzrbted and whuzzzshed and bzzaaped, etc, then it drew this:

Laurie was amazed. It was like magic, but every step made sense.

“So, knowing what the turtle can do, can you teach it how to draw a circle?” Tinker said.

“I don’t know,” Laurie said, “but I want to try!”

“That’s good enough for me. Here, you can work at my desk. There is plenty of paper and compasses and things like that.”

Laurie sat down at Tinker’s desk. She doodled with the compass and played with the turtle for a while, trying to remember what she knew about circles.

A circle is round. No, not just round, perfectly round. You put the pin in the center, and the pencil spins round. To make a bigger one you open the compass; to make a smaller one you close the compass. If you change the width of the compass when it’s spinning, it doesn’t make a circle...

Suddenly an idea, or maybe a memory, popped into her head: A circle is all of the points that are exactly the same distance from the center. Hmm... what if you

Go forward one inch,
make a mark,
go back one inch,
turn right a tiny bit,
then repeat!

She wound up the little turtle again and placed it on her poem. It buzzed and burbled for a moment, then drew this:

“It’s working!” she said. “Hey, it’s not stopping.” The turtle was drawing over dots it had already drawn.

“I think it’s because you told it to repeat, but not how many times,” said Tinker.

“Well, it should stop when the circle is done,” Laurie said.

“It doesn’t really understand circles,” Tinker said. “It’s just a toy turtle, remember? You have to teach it.”

Laurie thought a little more, then rewrote her poem:

CIRCLE:
Go forward one inch,
make a mark,
go back one inch,
turn right one degree,
repeat three hundred sixty times.

Then she realized that she could make circles any size she wanted. It was just like opening the compass wider:

TWO-CIRCLE:
Go forward two inches,
make a mark,
go back two inches,
turn right one degree,
repeat three hundred sixty times.

“This is interesting. You’re working really hard!” Tinker scratched his head. “But as it is, it’s no good.”

“Why?”

“People want to make lots of different circles,” he said. “I’ll have to keep a lot of algorithms of different sizes, just in case someone wants three-and-nine-thirteenths inches or four-and-three quarters.”

“Well, what if you tell the turtle how big to make the circle?” she said. “Maybe like this:”

ANY-CIRCLE (how-big?):
Go forward how-big inches,
make a mark,
go back how-big inches,
turn right one degree,
repeat three hundred sixty times.

“And then,” she said, “instead of ONE-CIRCLE or TWO-CIRCLE you can say ANY-CIRCLE(one), or two, or even one-and-eleventy-sevenths!”

“Good idea, Laurie. That’s a lot simpler.” said Tinker. “I was worried you were going to fill my shop with circles!”

“You know, the turtle is drawing really slowly. Not like when it was doing the square,” she said.

It was true. The turtle would crawl all the way to the edge of the circle, then make a mark, then crawl all the way back to the center, three hundred sixty times. With small circles it wasn’t too bad, but big circles took a lot longer.

“Hmm.” Tinker said. “It spends a lot more time running back and forth than it does making marks. Do you think you can reduce the running time?”

It makes sense, but it isn’t sensible. She thought & doodled and doodled & thought, but Laurie couldn’t figure out how to make it more sensible. The turtle has to go back to the center, right? How else could it know where the edge of the circle was?

Laurie let her eyes wander around the room. Xor was staring at a moth that was flying in lazy loops around a light bulb. His skin was slowly fading from red to yellow and back to red. The moth went round and round. It was hypnotic. Round and round and round and...

Oh.

She grabbed for a fresh piece of paper before the idea got away. Don’t let a new thing out of your sight without a name.

MOTH-CIRCLE (how-big?):
Go forward how-big inches,
make a mark,
turn right one degree,
repeat three hundred sixty times.

The turtle went bzzaap and zzzrbt and whuzzzsh and then it started to draw. It moved one inch, made a dot, then turned a tiny bit, then moved one inch, then made another dot...

“Whoops. It’s making a huge circle! Let me try a small number.” She didn’t have a small number handy, so she borrowed one she had heard from Tortoise: one thirty-second of an inch.

“That’s better.” Laurie said.

“Let me see,” Tinker said. “Wow, look at the little guy run!”

“That was fun,” said Laurie. “I didn’t know you could just make up new ways to do things.”

“Of course you can. Often you aren’t the first to think of something, but if it works, who cares? Now, for my end of the trade.”

“Did you find the shortest path?” Laurie asked.

“Not exactly. The bad news is that what you are trying to do is impossible.”

“It’s impossible?”

“Well, highly improbable. There are many different ways to visit all the towns. It seems like you could write an algorithm for the turtle to try each one and find the shortest, right?”

“Sure, why not?” said Laurie.

“There are twenty-one towns in Userland. How many paths do you think there are?” Tinker said.

“I don’t know,” said Laurie. “A hundred?”

“Way more.”

“Um, a million?” Laurie said.

“More like a million million times that!” said Tinker.

“But how can that be?”

“Let’s say there are only three towns: A, B, and C,” Tinker said. “You are already standing in A, so you have to worry only about B and C. How many ways can you go?”

“Well,” she said, “I could go from B to C, or go to C then B. That’s two.”

“That’s right! But BC is the same as CB, just backward. Every path has a mirror image, so with three towns there is really only one possible path that visits them all. What if there were four towns, A, B, C, and D?”

Laurie counted on her fingers. “I could go BCD, or BDC, or CBD, or CDB, or DCB, or... DBC. Six! No, three.”

“That’s three times as many. Add another town, twelve times as many,” Tinker said. “Add a sixth town and there are sixty different paths through all of them. With seven towns there are three hundred sixty paths. As you add more towns, the number of paths gets very big!”

3 Towns: 2  ÷  2 = 1
4 Towns: 2  ×  3  ÷  2 = 3
5 Towns: 2  ×  3  ×  4  ÷  2 = 12
6 Towns: 2  ×  3  ×  4  ×  5  ÷  2 = 60
7 Towns: 2  ×  3  ×  4  ×  5  ×  6  ÷  2 = 360
8 Towns: 2  ×  3  ×  4  ×  5  ×  6  ×  7  ÷  2 = 2,520
9 Towns: 2  ×  3  ×  4  ×  5  ×  6  ×  7  ×  8  ÷  2 = 20,160

“For twenty-one towns you have to multiply one times two times three times four, all the way up to twenty. It makes a HUGENORMOUS number!”

2  ×  3  ×  4  ×  5  ×  6  ×  7  ×  8  ×  9  ×  10  ×  11  ×  12  ×  13  ×  14  ×  15  ×  16  ×  17  ×  18  ×  19  ×  20  ÷  2 = 1,216,451,004,088,320,000

“!” said Laurie.

“Indeed!” Tinker said. “All of that ‘one times two times three’ stuff takes too long to write. So you can use the exclamation point as a shorthand.”

20!  ÷  2 = 1,216,451,004,088,320,000

“But that’s...” Laurie said, counting the commas, “over one million million million paths!”

“One of those umpty-million paths is the shortest,” Tinker said. “I don’t know of any way to find it quickly.”

“I’ll be old before we check them all! Isn’t there a better way?”

“Ah, that’s the good news!” Tinker said. “I only deal in Exact answers. But there is a brilliant Composer who lives in Permute, named Hugh Rustic. He deals in Good Enough answers. I send him all of my hardest cases. I’ll write an IOU that you can take to him.”

Like what you read? Lauren Ipsum is a children's story about computer science. Buy a copy and help us translate it into Spanish, Portuguese, and other languages.

A story about computer science
and other improbable things.

The Wandering Salesman is happy person who doesn't like to think too much. At every point he just goes to the nearest place he's never been to. This isn't a terrible way to do it. It's better than random but it's not terribly good. Below is an interactive game that shows how it works. Push the "Solve it!" button to see him run around. You can also move the cities around and push the button again to see how his path changes.

You might notice that his path often crosses itself. This is a sign of less than optimal solution. There are many (many) other algorithms to find more efficient paths. A reasonable optimization might be to look ahead, say, three steps, consider the lengths of all of the combinations, and choose the shortest.

My favorite is the one used by a later character named Hugh Rustic, which depends on an army of ants that walk randomly and communicate with each other using pheremones. I'm still learning these new-fangled graphics libraries but I'll write a post about that one soon.

A story about computer science
and other improbable things.

Below is a little game that lets you try to beat Jane yourself. The secret password is picked at random from a list of 10,000 dictionary words, and the box will show you valid words to make it a bit easier. No peeking at the source code. :)

(I'll give you a hint. Try ਲ.)

Spoilers below...

What you are doing is a timing attack. If checking a completely wrong password takes less time than a mostly-wrong password, you can use that information to guess the password, one letter at a time.

The attacker's strategy is pretty much the same as the one used in the children's game hangman. It's stuff like this that makes building secure systems very hard. Make just one mistake, and compromising your system becomes literal child's play.

To fix this particular information leak, Jane should take exactly the same amount of time to return her yes/no answer, regardless of how close the guess is to the password. For example, she can simply keep checking letters even if they are incorrect. Or she could use a stopwatch and always wait to give her answer until 30 seconds have passed. Checking passwords is one of the few cases where an algorithm must be made slower to work correctly.

Like what you read? Lauren Ipsum is a children's story about computer science. Buy a copy and help us translate it into Spanish, Portuguese, and other languages.

A story about computer science
and other improbable things.

I sometimes no longer ask this in job interviews, to see how candidates deal with probability. The answer relies on a simple but powerful idea: as long as the bias is constant, you can use it against itself. The same idea is used in bimetallic springs in mechanical clocks. The two different rates of expansion to keep the overall tension constant in the face of changing temperature.

Here are the four possible outcomes of two flips of a coin. You can click and drag the bias to see how the probabilities change. No matter the bias, the odds of getting a Heads then Tails is always exactly the same as getting Tails then Heads.

Suppose Heads comes up % of the time.

Heads	×	Heads	=
Heads	×	Tails	=
Tails	×	Heads	=
Tails	×	Tails	=

You'll need an average of flips to get a Fair flip.

The answer is to flip twice. If you get HH or TT, throw the result away and start over. If you get HT or TH, return the first flip as your result. This algorithm almost magically transmutes an unfair coin into a perfect, though inefficient, Fair Coin. This algorithm was first described by John von Neumann in 1951, and it's very likely happening inside your computer right now.

Like what you read? Lauren Ipsum is a children's story about computer science. Buy a copy and help us translate it into Spanish, Portuguese, and other languages.

A story about computer science
and other improbable things.

Ask any programmer and they will probably tell a story like this: "My {father, boyfriend, wife} is a very smart {lawyer, writer, geologist} and I tried to teach them programming, and it was a disaster. Most people just don't get it."

If you happen to get programming, you can glory in the feeling that you have a gift for understanding complexity. Otherwise, you can tell yourself that you just don't have the gift. It's no one's fault. It's how things are. You'll never be a musician or a ballerina either.

I don't buy that. It's like saying that anyone can learn to read but most people just don't get writing. There are too many useful ideas stuffed into the grab-bag we call "computer science" to keep locked away. More to the point: if so many smart, motivated people can't seem to wrap their heads around computer science, you have to wonder whether there is a problem with computer science.

Programming is hard. Logic is hard. And that's fine. But if it remains hard, if we leave this world just as complex as we found it, have we made real progress?

To the Romans, multiplying MCMIIX and LXI was a job for experts. Exponents were literally unthinkable. Today we expect better of six-year-olds. It's not that we're so much smarter. We discovered a better way to think about numbers. Is there a better way to think about programming?

To find out, a couple of years ago I started writing a children's story about programming. The goal was to find the bits I understood well enough to teach to a nine-year-old. After a few months I realized that I didn't want to teach programming as such. Learning how to program is a terribly frustrating experience.

When I was nine my school had a room full of donated Apple IIs. I remember a few things about it: the seemingly pointless exercise of "formatting" our disks, a boring lecture about search and replace, a silly song about the home row keys. Most of all I remember playing with Logo. Once the incomprehensible rituals of setting up the computer were satisfied, we got to play with a little turtle that could do anything --- provided you could figure out how to teach it.

I now know why all those rituals were necessary. What galls me is that, twenty-plus years later, pointless rituals haven't gone away. Most of what I've stuffed my brain with since then are the random things that must be done before real work begins.

The rest... the rest is the good stuff. They are both facts and habits of mind, and the best way to learn them is through play and discovery. The idea that most of what we call programming is just paperwork is not at all original to me. Alan Kay has been pounding this drum for longer than I've been alive. Seymour Papert invented Logo for precisely this reason.

Writing Lauren Ipsum has become a kind of Logo game: learning through learning how to teach. I've had to go back to the basics, to try to understand the why and how of things. I've learned that all integers are just zero and one in disguise. I finally learned that zero is an even number, and how to get perfectly fair flips out of any biased coin. Lewis Carroll's work is more subtle and profound than I ever imagined: "What the Tortoise Said To Achilles" is actually a proof of infinite regress buried in the foundations of logic itself.

In the coming months I will post more about the book and the ideas behind it as we get ready to publish. I hope you like it.

(Originally on Facebook's Engineering Blog.)


flic.kr/airfuel

Doppler is an internal project for mapping & measuring the network between users and Facebook. The data it collects has many uses, and it’s a good example of how we gather performance statistics at scale. [0]

Most Facebook users live thousands of miles away from our servers, so it’s useful to have a good idea about how their network path to us is behaving. If some DNS provider is not doing well in Europe, we want to know that. If some segment of users have low bandwidth or high latency, optimizing for network bytes rather than server CPU is a better use of our time.

Pinging the Vasty Deep

The Doppler probe is a 100-line JavaScript we append to a small random sample of Facebook page hits. The script initiates HTTP requests to some tiny images on our servers and times how long it takes to load them. We perform five to fifty million of these experiments every day. [1]

The first thing we started tracking was end-to-end DNS latency. To do this we set up a wildcard domain that answers to any hostname ending in dns.facebook.com. So you can hit, say, http://xyzzy.dns.facebook.com/favicon.ico and you’ll get an answer back.

Each time the Doppler JavaScript executes, it generates a random hostname prefix, constructs a unique URL, and times how long it takes to load it. Because the hostname and URL are unique, no caches exist anywhere and this experiment (A) encompasses a full end-to-end DNS lookup, then a TCP handshake, and then a minimal HTTP transaction. We then run experiment B, loading a second tiny image from the same hostname [2]. Since the DNS lookup already happened and has been cached by the user’s browser and operating system, experiment B is only a TCP connect and HTTP request.

Subtract B from A, and you get a measurement of the uncached, worst-case, end-to-end DNS latency of the given user. Early on we realized that measurement B by itself is roughly two full round trips over the network [3]. So we divide B by two and use that to track packet RTT between different points on the planet and our data centers. These measurements are not as accurate as you’d get from custom software running on both ends of the transaction, but they work well enough.

The Doppler JavaScript sends these measurements back to our servers via an AJAX request. On the server side we record other bits of metadata like the server name, user IP address, data center, timestamp, etc. Over the last year we've collected billions of network measurements along a half-dozen interesting dimensions. [4]

By geolocating the user's IP address and rolling up the records by country and data center, we get a measure of the network RTT and DNS latency from every country to every data center. We use those rollups to draw pretty graphs. When something big happens to the Internet (e.g., cable breaks), it manifests as jumps or dips in the graphs.

Another use of the country-level data is a "pessimizing proxy" called Netlab. From Palo Alto the site loads pretty fast, so our engineers don't feel what real users feel. With Netlab, we can simulate the Facebook experience of a user in other countries, using real-world numbers for packet latency and bandwidth. [5]

Country data is good for a high-level view. But political borders don't really match up with the real borders of the Internet. Countries can have many ISPs, and ISPs can span many countries. So we also annotate Doppler logs with the autonomous system number associated with the user's IP address. An autonomous system is kind of like an independent city-state on the Internet. There are about 35,000 of them, and they represent large networks of computers that are owned or controlled by the same entity. Facebook is an autonomous system, as are Google, large ISPs, carriers, and Harvard University. A packet may pass through many city-states on the way from here to there. If there is a network problem that throws country-level graphs way off, we use this dimension to narrow it down to a specific ISP or network.

We also use Doppler for evaluating TCP optimizations and new network hardware. We take a cluster of servers, apply the configuration or hardware, and watch the graphs to see if anything changes. For this we added two more experiments that load images of specific file sizes to measure our users' effective bandwidth.

Below is a graph of image download times from two of our VIPs. The blue line is the control, with no configuration changes. The yellow line is the experiment, which shows a big improvement:

This chart is also an example of why continuous measurements are a good idea. The configuration change was applied a couple of weeks before I heard about it. We could go back and see what effect it had.

Future Work

Recently we added support for mobile users. The measurement probe is written in JavaScript so it works fine on smartphones and a subset of feature phones. Mobile performance in general is in dire need of solid network data, and this is the simplest way to collect it at scale. For the same reason we'll soon add experiments for SSL and IPv6.

At the moment we are looking at using Doppler data for DNS-based global server load balancing. The basic idea behind global load balancing is that you have one domain name, e.g., www.facebook.com, but many data centers scattered around the world. The DNS load balancers are responsible for returning the address of the data center it thinks is physically closest to the user. The problem is that your DNS server doesn't talk directly to the user. It talks to intermediary resolvers that query on the user's behalf. So current systems return the data center physically closest to the resolver. This takes on faith that physical distance is a good substitute for Internet distance, and that the user is reasonably close to the resolver. When either of those assumptions are wrong, we're giving out the wrong answers.

Every large website has this problem. A while ago Google proposed an extension to the DNS protocol to include part of the user's IP address in the DNS lookup packet. Having a general idea of the ultimate user’s address would allow DNS load balancers to give out more intelligent answers. But convincing the whole world to adopt your protocol is hard. Fortunately, there is a cheap and cheerful way to do almost the same thing.

Remember that unique hostname, xyzzy? That can also be thought of as a unique ID for a Doppler measurement. Doppler actually results in two server-side logs. The first is the one we've been talking about, which contains the measurement ID, user IP, data center, and various measurements. The other comes from our DNS server. Normally, people don't log DNS traffic but it's easy enough to do. This second log contains the experiment’s ID paired with the resolver's IP address.

If you were to join those logs together on the ID, then roll up by data center and resolver IP, you get a direct measurement of Internet latency between your users and your data centers, but indexed by resolver. And that's what we're building right now: an alternate map of the world based not on geographic distance, but on pure network latency.

There are probably many more interesting things to do with this kind of data. If you run a website with a decent amount of traffic you can do stuff like this yourself. The important thing when designing your experiment is to keep it continuous, lightweight, and as broad as possible.

Many thanks to Eric Sung, Daniel Peek, David Recordon, Bill Fumerola, and Adam Lazur for their help with this article. Special thanks to Paddy Ganti and Hrishikesh Bhattacharya for their work on Doppler.

Notes

[0] People have been mapping & measuring the internet since virually the beginning. One of the earliest Internet mappers was created by Robert Morris in 1988. It exploited software bugs in other systems to self-replicate and report back what it saw. His program itself had a bug, which inadvertently crashed the Internet and introduced the term "computer virus" to popular culture.

A modern (non-destructive) longitudinal study of the Internet is CAIDA run by the Cooperative Association for Internet Data Analysis. They have been collecting data since 1997.

[1] Check out Yahoo's open source Boomerang measurement library. It includes plugins to measure DNS latency, network latency, bandwidth, and other cool stuff.

[2] It doesn't actually hit /favicon.ico, it hits an endpoint that returns all of the proper no-cache voodoo and closes the connection. Also, we use multiple domains to test SSL, DNSSEC, IPv6, etc.

[3] There are six packets involved in a minimal TCP handshake and HTTP request. As far as the client is concerned packet 6 is not part of the timing, and packets 3 & 4 are sent out at the same instant (sometimes as the same packet). So, if you squint hard enough, the time from the sending of packet 1 and the receipt of packet 5 (i.e., the time between setting the Image object's src attribute and its onload event) is equivalent to two network round-trips.

1. client SYN
2. server SYN + ACK
3. client ACK
4. client HTTP request
5. server HTTP response + FIN
6. client FIN + ACK

[4] Please note that we average thousands of samples together. Doppler is about gross Internet statistics, not per-user data.

[5] This is done by introducing packet delays and loss at the TCP level. You can do this yourself with Charles Proxy or dummynet. I highly recommend testing your software under high latency and narrow bandwidth.

(Excerpted from Lauren Ipsum, a book about computer science, programming, and other strange fantasies.)

“Is this the same place?” Laurie asked herself. “It certainly looks the same.”

She tried a right turn, but after a short while she was back where she started. When she tried a second time, and a third, and twenty-seventh time, she always came back to Recursion Junction.

“Every time I take a turn, it seems as though I am going somewhere else, but I always end up in the same place. What’s going on?”

She went round

...and round

...and round

...and round so many times that Laurie lost count. Just as she was about to give up, the next turn round put her on a different road.

The road was neat and straight, and seemed to stretch on forever. Ahead of her, a man in a Greek helmet was sitting on a large green round animal with a shell. They were moving slowly and steadily away.

“Hey! Wait!” Laurie shouted, running up to the pair.

“Ah, at last someone has caught up to us!” the animal said. “I thought it was impossible.”

“Don’t start THAT old argument again!” said the man.

“Hello, I am Tortoise, a humble tortoise,” said the animal. “And this is my esteemed companion, Achilles the Logician.”

“At your service, miss!” said Achilles, bowing grandly from his perch atop the Tortoise.

“Um, Hello. My name is Lauren Ipsum.” Laurie attempted a curtsey.

“‘Lauren Ipsum’. That’s quite a GENERIC name, isn’t it?” observed Achilles.

“Never mind that. How did you get here?” asked Tortoise.

“I don’t really know,” Laurie said. “I was following the path to Symbol but I got turned around at Recursion Junction.”

“That often happens. You spent a fair amount of time chasing your tail, I imagine.”

“But I don’t have a tail!”

“Got away from you, did it? Well, it should turn up again,” said Tortoise. “Or perhaps it was optimized away. But seeing as MOST of you is present, perhaps you can help us resolve a discussion.”

“Well, I can try.”

“Splendid. The question I was posing to my dear friend Achilles is this: How long is an infinite piece of string?”

“An infinite string? Infinite means it’s really really really really really REALLY long. Really.” said Laurie.

“Ah! You agree with ME,” Achilles said, “and so the burden of proof must be borne by the other side.”

“The burden of Achilles on my BACK is more than enough!” said Tortoise.

“My colleague the Tortoise is wise in many matters,” Achilles explained. “But he is clearly wrong this time. He maintains that an infinite string can be less than TWO INCHES long!”

“But how can an infinite string be two inches long?” Laurie asked.

“His claim sounds preposterous and indiscrete,” said Achilles. “We are in continuous disagreement.”

“I never disagree,” said Tortoise. “I only discuss, especially with a formidable intellect such as yours, Achilles.” Achilles preened at the Tortoise’s praise. “Allow me to suggest a way to settle the matter.”

“Please, suggest away,” said Achilles.

“Let us build —hypothetically, of course— an infinite piece of string, and then measure it. Laurie can be our impartial judge.”

“I accept. Experiment always beats Theory,” Achilles said. “And an impartial judge sounds wonderful, especially if she already agrees with me!”

“Excellent,” said Tortoise. “Let us begin. If you had an infinite number of pieces of string, and laid them end-to-end, would that be infintely long? Hypothetically?”

“Yes, it must be.” said Laurie.

“No matter how long or short each individual piece is?” asked Tortoise.

“Surely,” said Achilles. “Infinity is infinity.”

“I wonder. Suppose we start with a piece of string ONE inch long,” Tortoise said. “Then add a second piece of string that is ONE-HALF inch long. How long are they together?”

“One-and-a-half inches,” Laurie said.

“And that is shorter than two inches?” Tortoise asked.

“One-half inch shorter. Unmistakably.” Achilles answered.

“Laurie?”

“That sounds right.”

“We all agree thus far,” said Tortoise. “Perhaps we’ll converge on the same conclusion.”

“I doubt that!” said Achilles. Laurie wasn’t sure where Tortoise was going, but she doubted too.

“Achilles, would you please keep count of our hypothetical string? I want to add a third piece ONE-QUARTER of an inch long,” said Tortoise. “Is our string now one-and-three-quarters inches long?”

Achilles retrieved a much-used notebook from under his helmet and scribbled some figures. “It seems so,” he said.

“With one-quarter inch to spare?” asked Tortoise.

Scribble. “Yes, only one-quarter inch! You are a finger’s-width away from defeat!”

“Add an EIGHTH-inch piece,” Tortoise continued. “Do I still have some space left over?”

“Yes, but I’ll have beaten you soon!” Achilles crowed. “Your string is only an eighth-inch away from the limit, and you’ve only done FOUR pieces!”

“You may well prove right, Achilles, but honor demands that we continue until the bitter end,” said Tortoise.

“It won’t be long,” Achilles said graciously. “What is your next move?”

“I would like to add another piece of string, but this time one-sixteenth inches long.”

“Done!” Scribble. “You have only one-sixteenth inch left, old friend!”

“My word,” said Tortoise. “I should be careful with my allotted space. For the next one, I would like to add a piece of string one-thirty-second inches long.”

“As you wish, poor Tortoise, one-thirty-second inch added. Only one-thirty-second inch remaining, and an infinity of strings to go! There will be PLENTY of rope left over to hang yourself with!” said Achilles.

“Put on a sixty-fourth inch piece, a one-hundred-and-twenty-eighth inch piece, then a two-hundred- and-fifty-sixth inch piece, and a five-hundred-and-twelveth inch piece of string,” said Tortoise.

“Hold on, those are very big —no, SMALL— numbers!” Achilles figured and scribbled for a minute. “Ah! There is only a five-hundred-and-twelveth inch remaining! It’s too bad we’re not splitting HAIRS, or you could have gotten a little farther! Do you give up now?”

“Oh, wait, I see!” exclaimed Laurie. “Achilles, Tortoise is right.”

“What? Don’t change your mind NOW, when I am so close to victory!” Achilles cried.

“No, I’m sure Tortoise is right,” said Laurie. “Don’t you see? Every piece he adds is half as long as the one before. That leaves just enough room left over. Then he repeats again and again with shorter and shorter strings. Even if he adds an infinite number of pieces, it will NEVER reach two inches.”

“Well, hardly ever,” said Tortoise.

Achilles grimaced. “It seems you’ve done it again, Tortoise. But just to make sure, I will check the arithmetic MYSELF.” He proceeded to scribble in his notebook: one-thousand-twenty-fourth inches, plus two-thousand-forty-eighth inches, plus four-thousand-ninety-sixth inches, plus...

“THAT should keep him busy. Thank you for your assistance, Laurie.”

“You’re welcome, Mr Tortoise,” said Laurie. “I didn’t know something so big could also be so small.”

“Or that something so small can also be so big,” Tortoise said.

“Mr Tortoise, do you know how long this road is?” asked Laurie. “It feels like it goes on forever. I’m trying to get to the next town.”

“It’s quite long,” said Tortoise. “In fact, it is infinite.”

“Oh, no! How do I get to the end?”

“That can be done in two simple steps.”

“Really? What steps?”

“A step with your right foot, then a step with your left foot. But the attitude is important. It’s integral. Close your eyes and picture the road as only two steps long, like the string.”

Laurie closed her eyes, took a deep breath, and stepped forward with her right foot. Then she stepped again with her left foot. When she opened her eyes, Achilles and the Tortoise were gone. In front of her was a signpost:

A "full-stack programmer" is a generalist, someone who can create a non-trivial application by themselves. People who develop broad skills also tend to develop a good mental model of how different layers of a system behave. This turns out to be especially valuable for performance & optimization work. No one can know everything about everything, but you should be able to visualize what happens up and down the stack as an application does its thing. An application is shaped by the requirements of its data, and performance is shaped by how quickly hardware can throw data around.

Consider this harmless-looking SQL statement:

INSERT INTO some_table VALUES (1, 2, 3);

What is the maximum number of these you can do per second? Well, it first depends on the storage engine you use. If the table is in MyISAM format a write may result in one more disk-seek or rotation than InnoDB. InnoDB stores the row data right after the index. With MyISAM indexes and data are stored in different files. A hard drive can only do one thing at a time, so this detail can make a big difference in performance. It also depends on what mechanisms are in place (write-through caches, journals, etc) to keep random seeks to a minimum, and whether you are using table-level or row-level locking. Digging deeper into how these storage engines work, you can find ways to trade safety for speed.

The shape of the data

One way to visualize a system is how its data is shaped and how it flows. Here are a some useful factors to think about:

• Working data size: This is the amount of data a system has to deal with during normal operation. Often it is identical to the total data size minus things like old logs, backups, inactive accounts, etc. In time-based applications such as email or a news feed the working set can be much smaller than the total set. People rarely access messages more than a few weeks old.
• Average request size: How much data does one user transaction have to send over the network? How much data does the system have to touch in order to serve that request? A site with 1 million small pictures will behave differently from a site with 1,000 huge files, even if they have the same data size and number of users. Downloading a photo and running a web search involve similar-sized answers, but the amounts of data touched are very different.
• Request rate: How many transactions are expected per user per minute? How many concurrent users are there at peak (your busiest period)? In a search engine you may have 5 to 10 queries per user session. An online ebook reader might see constant but low volumes of traffic. A game may require multiple transactions per second per user.
• Mutation rate: This is a measure of how often data is added, deleted, and edited. A webmail system has a high add rate, a lower deletion rate, and an almost-zero edit rate. An auction system has ridiculously high rates for all three.
• Consistency: How quickly does a mutation have to spread through the system? For a keyword advertising bid, a few minutes might be acceptable. Trading systems have to reconcile in milliseconds. A comments system is generally expected to show new comments within a second or two.
• Locality: This has to do with the probability that a user will read item B if they read item A. Or to put it another way, what portion of the working set does one user session need access to? On one extreme you have search engines. A user might want to query bits from anywhere in the data set. In an email application, the user is guaranteed to only access their inbox. Knowing that a user session is restricted to a well-defined subset of the data allows you to shard it: users from India can be directed to servers in India.
• Computation: what kinds of math do you need to run on the data before it goes out? Can it be precomputed and cached? Are you doing intersections of large arrays? The classic flight search problem requires lots of computation over lots of data. A blog does not.
• Latency: How quickly are transactions supposed to return success or failure? Users seem to be ok with a flight search or a credit card transaction taking their time. A web search has to return within a few hundred milliseconds. A widget or API that outside systems depend on should return in 100 milliseconds or less. More important is to maintain application latency within a narrow band. It is worse to answer 90% of queries in 0.1 seconds and the rest in 2 seconds, rather than all requests in 0.2 seconds.
• Contention: What are the fundamental bottlenecks? A pizza shop's fundamental bottleneck is the size of its oven. An application that serves random numbers will be limited by how many random-number generators it can employ. An application with strict consistency requirements and a high mutation rate might be limited by lock contention. Needless to say, the more parallelizability and the less contention, the better.

This model can be applied to a system as a whole or to a particular feature like a search page or home page. It's rare that all of the factors stand out for a particular application; usually it's 2 or 3. A good example is ReCAPTCHA. It generates a random pair of images, presents them to the user, and verifies whether the user spelled the words in the images correctly. The working set of data is small enough to fit in RAM, there is minimal computation, a low mutation rate, low per-user request rate, great locality, but very strict latency requirements. I'm told that ReCAPTCHA's request latency (minus network latency) is less than a millisecond.

A horribly oversimplified model of computation

How an application is implemented depends on how real computers handle data. A computer really does only two things: read data and write data. Now that CPU cycles are so fast and cheap, performance is a function of how fast it can read or write, and how much data it must move around to accomplish a given task. For historical reasons we draw a line at operations over data on the CPU or in memory and call that "CPU time". Operations that deal with storage or network are lumped under "I/O wait". This is terrible because it doesn't distinguish between a CPU that's doing a lot of work, and a CPU that's waiting for data to be fetched into its cache.[0] A modern server works with five kinds of input/output, each one slower but with more capacity than the next:

• Registers & CPU cache (1 nanosecond): These are small, expensive and very fast memory slots. Memory controllers try mightily to keep this space populated with the data the CPU needs. A cache miss means a 100X speed penalty. Even with a 95% hit rate, CPU cache misses waste half the time.
• Main memory (10^2 nanoseconds): If your computer was an office, RAM would be the desk scattered with manuals and scraps of paper. The kernel is there, reserving Papal land-grant-sized chunks of memory for its own mysterious purposes. So are the programs that are either running or waiting to run, network packets you are receiving, data the kernel thinks it's going to need, and (if you want your program to run fast) your working set. RAM is hundreds of times slower than a register but still orders of magnitude faster than anything else. That's why server people go to such lengths to jam more and more RAM in.
• Solid-state drive (10^5 nanoseconds): SSDs can greatly improve the performance of systems with working sets too large to fit into main memory. Being "only" one thousand times slower than RAM, solid-state devices can be used as ersatz memory. It will take a few more years for SSDs to replace magnetic disks. And then we'll have to rewrite software tuned for the RAM / magnetic gap and not for the new reality.
• Magnetic disk (10^7 nanoseconds): Magnetic storage can handle large, contiguous streams of data very well. Random disk access is what kills performance. The latency gap between RAM and magnetic disks is so great that it's hard to overstate its importance. It's like the difference between having a dollar in your wallet and having your mom send you a dollar in the mail. The other important fact is that access time varies wildly. You can get at any part of RAM or SSD in about the same time, but a hard disk has a physical metal arm that swings around to reach the right part of the magnetic platter.
• Network (10^6 to 10^9 nanoseconds): Other computers. Unless you control that computer too, and it's less than a hundred feet away, network calls should be a last resort.

Trust, but verify

The software stack your application runs on is well aware of the memory/disk speed gap, and does its best to juggle things around such that the most-used data stays in RAM. Unfortunately, different layers of the stack can disagree about how best to do that, and often fight each other pointlessly. My advice is to trust the kernel and keep things simple. If you must trust something else, trust the database and tell the kernel to get out of the way.

Thumbs and envelopes

I'm using approximate powers-of-ten here to make the mental arithmetic easier. The actual numbers are less neat. When dealing with very large or very small numbers it's important to get the number of zeros right quickly, and only then sweat the details. Precise, unwieldy numbers usually don't help in the early stages of analysis. [1]

Suppose you have ten million (10^7) users, each with 10MB (10^7) bytes of data, and your network uplink can handle 100 megabits (10^7 bytes) per second. How long will it take to copy that data to another location over the internet? Hmm, that would be 10^7 seconds, or about 4 months: not great, but close to reasonable. You could use compression and multiple uplinks to bring the transfer time down to, say, a week. If the approximate answer had been not 4 but 400 months, you'd quickly drop the copy-over-the-internet idea and look for another answer.

movies.example.com

So can we use this model to identify the performance gotchas of an application? Let's say we want to build a movies-on-demand service like Netflix or Hulu. Videos are professionally produced and 20 and 200 minutes long. You want to support a library of 100,000 (10^5) films and 10^5 concurrent users. For simplicity's sake we'll consider only the actual watching of movies and disregard browsing the website, video encoding, user comments & ratings, logs analysis, etc.

• Working data size: The average video is 40 minutes long, and the bitrate is 300kbps. 40 * 60 * 300,000 / 8 is about 10^8 bytes. Times 10^5 videos means that your total working set is 10^13 bytes, or 10TB.
• Average request size: A video stream session will transfer somewhere between 10^7 and 10^9 bytes. In Part One we won't be discussing networking issues, but if we were this would be cause for alarm.
• Request rate: Fairly low, though the concurrent requests will be high. Users should have short bursts of browsing and long periods of streaming.
• Mutation rate: Nearly nil.
• Consistency: Unimportant except for user data. It would be nice to keep track of what place they were in a movie and zip back to that, but that can be handled lazily (eg in a client-side cookie).
• Locality: Any user can view any movie. You will have the opposite problem of many users accessing the same movie.
• Computation: If you do it right, computation should be minimal. DRM or on-the-fly encoding might eat up cycles.
• Latency: This is an interesting one. The worst case is channel surfing. In real-world movie services you may have noticed that switching streams or skipping around within one video takes a second or two in the average case. That's at the edge of user acceptability.
• Contention: How many CPU threads do you need to serve 100,000 video streams? How much data can one server push out? Why do real-world services seem to have this large skipping delay? When multiple highly successful implementations seem to have the same limitation, that's a strong sign of a fundamental bottleneck.

It's possible to build a single server that holds 10TB of data, but what about throughput? A hundred thousand streams at 300kbps (10^5 * 3 * 10^5) is 30 gigabits per second (3 * 10^10). Let's say that one server can push out 500mbps in the happy case. You'll need at least 60 servers to support 30gbps. That implies about 2,000 concurrent streams per server, which sounds almost reasonable. These guesses may be off by a factor or 2 or 4 but we're in the ballpark.

You could store a copy of the entire 10TB library on each server, but that's kind of expensive. You probably want either:

• A set of origin servers and a set of streaming servers. The origins are loaded with disks. The streamers are loaded with RAM. When a request comes in for a video, the streamer first checks to see if it has a local cache. If not, it contacts the origins and reads it from there.
• A system where each video is copied to only a few servers and requests are routed to them. This might have problems with unbalanced traffic.

An unknown factor is the distribution of popularity of your video data. If everyone watches the same 2GB video, you could just load the whole file into the RAM of each video server. On the other extreme, if 100,000 users each view 100,000 different videos, you'd need a lot of independent spindles or SSDs to keep up with the concurrent reads. In practice, your traffic will probably follow some kind of power-law distribution in which the most popular video has X users, the second-most has 0.5X users, the third-most 0.33X users, and so on. On one hand that's good; the bulk of your throughput will be served hot from RAM. On the other hand that's bad, because the rest of the requests will be served from cold storage. A useful hack might be to keep the first minute of every video in RAM at all times.

Whatever architecture you use, it looks as though the performance of movies.example.com will depend almost completely on the number of concurrent reads your storage devices can handle. If I were building this today I would give both SSDs and non-standard data prefetching strategies a serious look.

It's been fun

This subject is way too large for a short writeup to do it justice. But absurd simplifications can be useful as long as you have an understanding of the big picture: an application's requirements are shaped by the data, and implementations are shaped by the hardware's ability to move data. Underneath every simple abstraction is a world of details and cleverness. The purpose of the big fuzzy picture is to point you where to start digging.

Notes

[0] Fortunately there is a newish tool for Linux called "perf counters".

[1] Jeff Dean of Google deserves a lot of credit for popularizing the "numbers you should know" approach to performance and systems work. As my colleague Keith Adams put it, "The ability to quickly discard bad solutions, without actually building them, is a lot of what good systems programming is. Some of that is instinct, some experience, but a lot of it is algebra."

The future is a disagreement with the past about what is and is not important.

The differences between two ages are informed by politics, technology, demographics, etc. But they are easiest to understand in terms of what each age thinks is worth caring about.

Knowing who your parents are is less important today than a couple hundred years ago. It doesn't cripple you socially as it once did. Knowing what material your water pipes are made of became suddenly important when we noticed the crippling effects of lead. Today we pay close attention to lab tests and labels and indicator lights that summarize what's really going on, because a lot of important stuff is too small or complex to comprehend directly. If I had to pick a major disagreement between the present world and the past, it would be the importance of invisible amounts of mass and energy, be they trace chemicals or transistors. Moreover we tend to care about emergent information content, the patterns in the material, rather than the actual material.

To a typical Victorian that wouldn't be heresy so much as fantastic nonsense. Your great-grandparents' world was populated by people, animals, and human-scale artifacts. Man was more literally the measure of all things. Important things were assumed to be big and obvious, or at least visible to the senses. Germ theory was one of the first ideas to break that assumption in a serious way. It didn't help that its companion idea, vaccination, was even weirder.

The point is that if you want to do truly "futuristic" work, you can't just extrapolate from what is believed right now. You have to suppose that at least one aspect of our worldview is wrong: something we hold dear is not actually important, or there is something else that should be, or both. If you imagine people feeling and acting exactly the same way as they do now, that's not the future. That's just later on. Also remember that the future is not an ever-upwards spiral. Who would have predicted in 1930 that mass slavery would return to Europe?

The flip side is that these discontinuities make predicting the future hard. The clues to how we may think are camouflaged among thickets of established fact. You have to isolate assumptions underneath our view of the world, alter them, then look around to see what changed. The deeper and more unspoken the assumption, the greater the potential for change. On the surface it sounds like a pretty stupid way to spend your time: disassembling ideas that aren't broken in order to discover ways to break them. On the other hand, that's precisely what tinkers do with machines: break them in order to understand and improve on them. That is, I think, what is meant by the motto "the best way to predict the future is to invent it". The initial spadework is the same whether you are predicting or inventing.

You needn't court-martial everything you think you know, but you do need ways to identify suspicious areas, and tools to dig deeper. So what might they be? There is one that I was mighty proud of inventing, until I described it to a Literature major.

Popular culture as a lens

One way to find these unspoken assumptions is to examine the treatment of a futuristic subject in popular culture, pick out the common themes, and ask whether they make sense. Popular culture is pretty good for this kind of practice, though what you get out of it may not be earth-shattering. A fiction writer's stock-in-trade is ideas that feel right but are not necessarily backed up by evidence.

For example, science fiction overflows with stories about artificial beings. What do they have in common? Well, a common trope is that artificial intelligences are sufficiently human de facto, especially if the story's conflict is about their status de jure. It's rare to find a story about AI which concludes that they are utterly and forever alien. Wintermute employed synthetic emissaries, and spoke casually about its motivations and desires. Agent Smith is explicitly Neo's mirror image. HAL9000's voice was as warm and soothing as a late-night radio host's. The golem of Jewish folklore is an interesting case. They are humanoid but explicitly not intelligent. Their moral position is somewhere between djinns and power tools. Terry Pratchett explored the idea of intelligent golems in "Feet of Clay", and it turned out exactly as you expect it would: after many misunderstandings they assert their rights and join the mainstream of society. It's hard to shake the feeling that AI stories are mostly allegories for racial integration.

We seem to have a hard time coming to grips with intelligence that does not have a face or a voice. When we say "intelligent" in casual speech, we mostly mean "a bloke I can have a conversation with". It may turn out that artificial intelligences are not able to evoke empathy from or experience empathy for natural born humans. I'm not sure I like the idea of what happens when we compete with such beings for resources.

The phrase "artificial intelligence" itself may harbor simplistic assumptions, like calling an X-ray machine a "magic lantern". It's not a bad analogy, but it misses important things like how X-ray waves are generated, other species of exotic radition, and how too much of them will kill you. Imagine a passive fabric of knowledge that, when and only when directed by a human, accomplishes superhuman feats. The conflict would not be over the humanity of the entity in question, but over which humans control it.

So, an unoriginal though interesting prediction: the idea of artificial intelligence having a distinct personality with recognizable motivations and desires is attractive, but there is little evidence that it must happen.

Subtextual subversion

Fans of hard science like to deride Derrida for being pseudo-intellectual, but this literary method is more or less what he was talking about. My wife was very pleased to point that out when I tried to pass it off as my own invention. Deconstructionism got lost in the weeds because they don't use reality to verify their theories, but the basic method seems sound. Can you use it on other bodies of literature, not just fiction?

Over the last ten years, the field of data-mining shifted its focus to gathering enough of the right data instead of just ever more-clever algorithms. I remember reading a lot of papers on automatic text classification in the late 1990s. They drew from a small pool of datasets, such as a collection of news articles. Innovation happened in the algorithms. This was a reasonable idea. Data was hard to come by, and using the same datasets seemed like a good way to compare the performance of different algorithms. The underlying assumption was carried over from other fields of computer science: given a representative sample of data, the way forward is to come up with more sophisticated algorithms.

Researchers at Google were the first I know of to demonstrate value in the opposite: dumb algorithms executed over gigantic datasets. They came to this opinion because they had so much damned data that it was hard enough to count it, much less run O(n³) algorithms over it. So they tried dumb algorithms first, and they worked surprisingly well. Older, naive algorithms turned out to be perfectly valid; they just needed orders of magnitude more input than had been previously tried. I would not be surprised to learn that several people thought of this idea early on. I don't know enough about the field to say.

It's possible you would have hit on the same idea, if you'd analyzed the literature for unspoken assumptions. Or, like Google, you could have played with big problems and new technology while under the gun to produce something useful.

Adopt early and often

Rubbing up against the new is another way to glimpse the future. Just as a child is immersed in a culture and then later derives the rules which shape it, early adopters immerse themselves in new ideas and new technology in order to puzzle out the shape of the future.

Some people are natural early adopters. A friend of mine is busy building an electronic library and giving away his physical books. In eight months he burned through two Kindles and an iPad, and I added a lot of his books to my shelves. The funny thing is that both of us genuinely feel we're getting a good deal: he is divesting his burden of dead trees and space, and I am saving perfectly good books from futuristic folly.

Ours is a classic future/past disagreement. He thinks it's better to move to this new new thing and see how it works. Eventually books will be published on-demand and kept up-to-date just as websites are. Paper will be an option, and not the most popular one. I think that paper is actually a pretty good medium for archival storage. Individuals should act in concert to preserve as much as we can, as more and more of our culture becomes digital-only. I don't know which of us will be right, or both, or neither.

My friend's way to predict the future is to surround himself with new technologies. The new often embodies upcoming disagreements with the present. You still have to do the work of isolating the assumptions it breaks, and deciding whether they are correct. For whatever reason I don't have the temperament for this. My method is to surround myself with early adopters, and watch what they do.

Bring it home

Picture yourself as you were ten years ago. List five things that are different about you now. I'll bet money that most of them are differences in your attitude towards the world. Now picture yourself ten years from today. You probably imagine external qualities: you will be more sucessful, or relaxed, or in Alaska. But most likely the biggest differences will be internal, what Future You thinks is truly important. It's hard to predict exactly what would change. If you knew that, you'd already be on your way to becoming the Future You.

If you wanted to preserve important bits of our civilization for future centuries, you could do worse than a bundle of paper sealed in plastic. It's remarkably cheap and effective; you can make one over a weekend. In this article we'll build a 1/2 scale model of a time capsule that contains the complete Linux 0.1 source code, plus sundry articles and internet ephemera.

A time capsule must perform three basic functions:

1. Encode information with sufficient density & durability.
2. Protect the information from physical damage, moisture, heat & cold, etc.
3. Be findable.

So while the Rosetta Stone performed (2) fairly well, it was pretty lucky to be found at all. Also its data density is terrible: about 1 bit per cubic centimeter. A book in a library fulfills (1), but requires the library around it to provide (2) and (3).

The internet, contrary to popular belief, is not very good at preserving information on a long time scale. It ultimately depends on digital media that break down rapidly. Early Unix source code, one of the most important sequences of bits ever written, had to be reconstructed from printouts.

The makeup of our capsule is simple: cellulose, carbon, polymers, and distributed information. You print a bundle of paper, place it inside a box, stick a label on it, then drown it in translucent epoxy resin. Alongside whatever it is you are preserving, you include the locations of other capsules.

Density & Durability

The humble piece of paper has come a long way in the last few decades. Acid-free paper [0] is the norm. It has archival properties comparable to cotton rag or parchment, and can easily survive for 300 to 500 years. Black & white laser toner is carbon powder and resin, fused to the surface at a few hundred degrees Fahrenheit. Carbon, being an atomic element, never fades. All in all it's a cheap & cheerful way to preserve data for a very long time.

Protection from the elements

You need an airtight seal that is itself fairly rugged. Epoxy resin is the hard plastic you often see protecting the surface of bars and restaurant tables. It's the closest thing you can get to man-made amber. Our scale-model capsule will be encased in a shell of resin about 1 centimeter thick. We're not shooting for 2,000 years exposed in the desert, just 50 to 500 years in the ground.

Be findable

The biggest design problem of traditional time capsules is that people forget where the damned things are buried. There seem to be two contradictory thoughts going on at once: that the best way to preserve information is inside a buried box, but that the best way to preserve information about the box is somewhere else.

Inside each time capsule will be a list of other known capsules. That, I hope, will make the difference between a node in a network and a forgotten box of junk. Dozens or hundreds of people could build full-scale capsules like this and share location data with each other. This prototype and its twin are the only two of their kind so far, so they only link to each other. The larger the network, the greater the chances of recovery.

Materials

1. 70 sheets of high-quality, acid-free printer paper
2. Laser printer
3. 250 pages of data you want to preserve
4. Scissors or paper cutter
5. Masking tape
6. 1/4" thick balsa wood planks
7. Illustation board (thick paperboard)
8. Razor, ruler
9. Wood glue or white glue
10. 500ml clear epoxy resin
11. Disposable cups and stir sticks
12. Gloves, goggles, and mask

The Bundle

Gather whatever data you want to preserve. It can be books, songs, computer programs, your Facebook page, diary, recipes, anything. I would focus on things that are likely to disappear. The future will probably most appreciate a description of boring, everyday life in Right Now, A.D.

Laser-print your data "4up" and single-sided. You should experiment with your printer's capabilities, but I've found that 10pt Helvetica printed 4up is the smallest mine can go and still be legible. Don't print double-sided because the toner might stick to itself if it ever gets too hot.

Cut the sheets into quarter-pages, collate them, then tightly wrap the bundle in a couple of layers of paper, like a Christmas present.

The Box

The box is mainly for appearance's sake, and to protect the paper from light. You could probably sink your bundle (wrapped in a few layers of paper and plastic) directly into the resin and it would work fine.

I made mine from illustration board. Cut two 12 x 15cm pieces for the top and bottom. Then cut the side walls 3cm high and slightly shorter than the 12 or 15 cm, to account for the thickness of the neighboring wall.

Glue it up! It doesn't have to be pretty or perfect as long as it fits as tightly as possible around the paper bundle. Let it sit for an hour to dry.

Place the bundle inside a ziplock bag, squeeze all the air out, and seal it. Put that inside the box and glue the lid shut. Paint it if you want, then glue a label to the front so people know what it is.

You build the mould the same way as the box, just 2cm larger in each dimension. I built mine out of balsa wood. If you have an aluminium or plastic tray of the right dimensions you can use that instead.

The Epoxy

Needless to say, do all epoxy work in a well-ventilated area and follow the safety instructions. Epoxy resin sounds tricky but it's pretty easy to handle with practice. There are many types of resin of varying properties. You want "encapsulating epoxy resin" or "clear casting resin", which is often used to seal electronic components and art projects. The strongest resins take 48 hours to harden completely, but last much longer than fast-cure resins.

Mix & pour about 3/4 cup (140ml) of resin in the bottom of the mould and let it cure for about an hour. This forms the back of the shell.

Center the box inside the mould. Mix & pour the rest of the resin on top of the box, and let it flow into the sides. Loosely cover and let cure for 24 to 48 hours. Your inner box will probably not be water-tight, so expect some bubbles to stream out of it as the epoxy seeps in. (To avoid this you could use an airtight tin, though there is a chance it will float!)

Place in ground, let stand 300 years.

Thinking Bigger

This is a scale model to demonstrate the process. Real capsules will contain at least 500 full-sized sheets of paper. The magic of the square-cube law makes it more cost-effective as you scale up. Casting large volumes of epoxy is a bit tricky, so start small and ask your friendly neighborhood supplier for advice.

Three Reams (1,500 sheets): This is probably the most managable size for a weekend project. You could use ready-made "archival" boxes for the inner box and one of those plastic file-folder boxes for the outer mould. Artist-grade resin starts to get pretty expensive at these volumes, but you can use amber encapsulating resin instead, about $80 for two gallons. AeroMarine sells in bulk and will send you free samples.

Carton (5,000 sheets): If you want something with more volume and durability, you can use a concrete flower pot whose inner dimensions are about 2 cm larger than the outer dimensions of the paper carton. Tightly seal the paper with several layers of thick plastic, and pour the resin as before.

Oil Drum (28,000 sheets): if you have good concrete molding skills and access to bituminous resins (instead of expoy resin) for the water seal, you could build a time capsule around a 55-gallon oil drum with very good capacity. It's probably wise to invest in higher-quality printing at this scale, so you can fit more than 4 pages of data per sheet.

Monument (3,500,000 sheets): A typical twenty-foot cargo container can hold over 700 cartons of paper. Constructing this monument requires a proper concrete foundation, a steel-reinforced concrete shell, and serious seals against moisture. The curation and printing alone are jobs of unusual size, but doable.

A cargo container sells for about USD$3,000. A contractor friend of mine estimated the concrete construction work would cost about USD$15,000. The printing, curation of the data, and mosaic work would be the most expensive items. But I believe the total cost would be under USD$100,000. That's less than many corporate sculptures, and a lot more useful.

I think it would be beautiful to put giant concrete archives in public parks around the world. A mosaic on the top surface would describe what it is and what it's for. Sink them about 2 meters down so they stick out a bit. They would form large benches for people to sit and play on, trace out the mosaic with their fingers, and perhaps be reminded of time.

If you want to build a time capsule yourself, send me an email! Let's get this thing started.

Notes

[0] "Acid-free" is a bit misleading. All wood-pulp paper contains acid that will yellow and destroy the fibers over time. "Acid-free" paper is given an extra wash, then impregnated with alkalies (baking soda, more or less) to improve whiteness and neutralize remaining acids. The percentage varies from 2% by weight up to 4 or 5% for "archival" quality paper.

This is a map of the global telegraph network from 110 years ago side-by-side with the internet of today:

One way to see the internet is as a physical manifestation of trade volume between cities, on a 40-year moving average. That is about how long it takes for economic ties to develop, demand to rise, and high-volume communications routes to be financed and built. Once built, these links tend to stick around.

Governments and empires have come and gone, bandwidth has increased a billion-fold, but the network has the same general shape it had back when Mark Twain was sending witty telegrams. The only big change since then is greater ties between the US and Asia.

Just from looking at where the cables go you can guess how long it would take to send a message. A telegram from San Francisco to Hong Kong in 1901 must have taken many hops through British Empire cables to Europe, through the Middle East, and so on. London to New York was fast and direct. The vestiges of the Spanish and Portuguese Empires show up in the many links between South America, the Caribbean archipelago, and the Iberian peninsula.

A cool thing is that you can measure these relative latencies yourself, using the present-day internet. If you run a website with a decent amount of worldwide traffic, you can use that traffic to map out how the internet responds with regards to you, and see how that matches with the gross structure of the 'net.

I wrote about a cheap and cheerful way to generate this data last year, and the code has since been open-sourced as part of Yahoo's Boomerang measurement framework. The basic idea is to have your users perform two tiny network requests: one to a throwaway hostname generated in the moment, like 8j48sas.dns.example.net/A.gif, then another to a different single-pixel image on the same host, 8j48sas.dns.example.net/B.gif. The first request will require a DNS lookup, TCP handshake, and HTTP transaction. The second only needs to do the TCP and HTTP steps. Now you have fuzzy measurements of how long it took to do a full HTTP round-trip (B) and to do a full end-to-end DNS lookup (A - B).

Real-world data on DNS performance is generally considered hard to come by. The domain name system is designed with caching and intermediaries at all levels, so you as a site owner only see part of the story during normal operation. You can buy precise data from commercial services like Gomez or Keynote. Or you generate it yourself if you happen to have, say, a distributed network of computers, or a browser plugin installed on millions of clients. Otherwise, this Javascript method is less accurate but works well enough.

Here is a chart of median (50th percentile) DNS latencies experienced by a random sample of Facebook users, broken down by country. As you can see, there are several lines crowding together at the bottom. That is the US and parts of Europe like the UK and Belgium. Facebook's DNS servers tend to be physically close to users in those countries. Spain and France are a bit higher up, and the rest of the graph is a mix of Asian and South American countries. [1]

The median value only tells part of the story. Here is the worldwide DNS latency data as a density plot, to show the distribution. Notice that a substantial number of users took more than 500 milliseconds just to look up a hostname. This is the uncached worst-case, of course, but it's something to keep in mind.

HTTP Latencies

Here is the chart for measurement B, the TCP + HTTP latency. This better reflects the real "geography" of the internet, because the HTTP requests travel all of the way back to our web tiers in the United States. There is much less volatility in these measurements day-to-day; it's controlled more by basic network conditions and speed-of-light and less by the health of various DNS recursors around the world.

How low can you go?

So how fast are these links between countries, compared to what is possible? Below is a chart of the same median HTTP latency data, averaged over a week. The short light-grey bars represent the theoretical minimum. If you could carve a direct line between any two spots along the surface of the planet, this grey bar would be the internet round trip time between the US and the given country. [2]

We can learn a lot of things from this chart. The most obvious is that HTTP latency between Asia and the US is worse than US-Europe. The Pacific Ocean is wider than the Atlantic, of course, but raw distance is not the only factor. Economics and local geography play their part.

Look at the ratios between the black bars (real) and the grey bars (theoretical). Both the fastest European and Asian countries have real-world latencies at or below 2X the theoretical minimum, which is pretty impressive. Few technologies get within spitting distance of the physical limits of the universe.

These low-multiple countries tend to have fortunate geography, or a strong history of economic relations with the United States, or both. Other countries with less-strong trade ties, such as Spain, or lots of little islands like the Philippines, have multiples nearer to 2.5X and above. While Australia is a bit farther than Thailand it's 15% closer as far as the internet is concerned. More investment has been put in by the cable operators to make that route fast and wide. In fact, Australia (population 22M) a comparable amount of bandwidth to the US as all of South America (population 385M).

Cry for Argentina

The multiples of South American countries start at 3.5X and go up from there. North-South routes are hurt by an unlucky trifecta of mountains, long land crossings, and archipelagos. There is only one cable that serves the Pacific side from Los Angeles to Panama. It's hard to justify building lots of capacity on the Pacific side, because the Andes mountains cut off that part of the continent from the rest. Most traffic follows a long and painful path across the entire length of the US to the Atlantic, then takes a right turn and down another 800 miles of the Florida peninsula. It exits Miami and immediately hits a congested maze of cables, hopping in and out of the water as it navigates the islands of the Caribbean. Someday South America will get better connected, but natural barriers drive the costs way up.

There are other interesting cases such as Belgium, which has the lowest latency and lowest multiple (1.6X) of any European country. The reason is that Belgium is well-placed as an internet nexus, being a) close to Britain but away from the Channel and b) geographically convenient for branching off into the rest of Europe.

Try this at home

These measurements are very skewed towards the United States. It would be awesome to see measurements from other spots and different traffic patterns from around the world. The code to collect this data (and a lot more) is open-source and simple to implement. So try the experiment for yourself and let me know what you find.

Notes

[1] This chart generally agrees with data gathered by Yahoo and Microsoft. The data is very US-centric; the picture will be quite different if you were to run the experiment from a site based on another continent. Facebook's servers are largely in the US, so naturally we care most about how to get bits from here to there and less about, say, between India and Saudi Arabia.

[2] The theoretical minimum latency is calculated using the average speed of light through optical fiber, over a hypothetical cable laid in a great circle line between the town of Independence, Kansas and the centroid of the given country. This time is multiplied by 4 to approximate the two round-trips necessary to complete a TCP handshake and HTTP transaction. You can read all about Great Circle routes and the speed of light through fiber in Wikipedia, or just use Wolfram Alpha to do it for you.

Alan Kay has a classic talk about different modes of thinking and an experiment he did with young people of different ages. Imagine a robot that can do four things: move forward some distance, move backward, rotate some number of degrees, and make a mark on the floor. The goal is to write a program that tells the robot to draw a circle.

Before I tell you what happened, do take a minute and try it yourself. It's a pretty problem.

Precocious teenagers are stuffed full of facts about circles and curves, πr² and calculus and all of that. More to the point, they have been trained in school to think abstractly and to leave aside visual or visceral thinking. Given this robot exercise they would spend 45 minutes on the blackboard writing equations and looking up things in books and scratching their heads before giving up.

Then Kay found some 10-year-olds and gave them the same problem. A 10-year-old doesn't know much about equations. She's a visual thinker. She doesn't know much about circles either, but she does know that a circle is defined as all points that are the same distance from the center. After about ten minutes of thinking and doodling she writes the program this way:

1. Start at center.
2. Move forward N inches.
3. Make a mark.
4. Move backward N inches.
5. Rotate 1 degree.
6. GOTO 1

That is a significant result. Ten-year-olds beat the pants off teenagers by using an officially discouraged mode of thinking. Not satisfied with that, crazy old Kay invited in a bunch of five-year-olds. You don't let little kids play with chalk if you want to get any work done, so for them he turned it into a game:

"Ok kids, you are robots. Say Beep beep beep! Good, now cover your eyes, and I want you to walk in a circle."

The kids would cover their eyes and cock their heads for a few seconds, thinking about how to walk in a circle while blind. Then they did it. They walked forward just a little, then turned a little, then walked forward again, and so on until their inner ear told them they'd done a full circuit. Not only did these children solve the problem immediately using visceral thinking, their solution was hundreds of times more efficient. Somehow they were intuiting differential equations.

What can I teach a child about computer science? It may be better first to ask what a child can teach me. I don't mean that in the Romantic sense of noble savagery or the wisdom of innocence. Computer science is the study of process: how to do things, how to figure out things, and especially how to figure out how to do things. We have to investigate everything that seems to work, wherever we find it. Visually and viscerally-centered thinking seem like rich veins to explore.

One of Alan Kay's favorite sayings is that having the right point of view is worth 80 IQ points. No matter how clever you are, if you're trying to figure out cube roots on an abacus you are not going to get very far. The right point of view can make the difference between an elegant solution and a pile of steampunk.

Fortunately it cuts both ways. If something you are working on seems to be inelegant, too full of random details and not enough clear principles, that's a sign you might be approaching it with the wrong point of view. Being able to recognize that state and pull yourself out of it is an invaluable skill. Both the hidebound and the feckless are crippled because they can't choose where their minds go next. An interesting, perhaps defining, characteristic of highly creative people is their ability to shift their point of view at will.

Often, especially when learning some large new system, programmers get stuck on something that looks like a bug and ask someone for help. Halfway through explaining their problem they realize where they had gone wrong. Never mind, never mind, it's not a bug, I'm an idiot. At a company I used to work for we had a stuffed bear called the bugbear. Before you could bug anyone else about your bug, you had explain it to the bugbear. Just thinking to yourself doesn't always work; there is something about vocalizing your mini-mental crisis to another being that engages the brain.

So, back to the question: what can we teach a child about computer science? It isn't an abstract set of laws you learn because it's good for you. Computer science is the study of how to do things --anything-- faster and better. It's more than that, of course. But the practical side is ultimately the source and purpose of all this commotion. (Right?)

• Two people painting a house is task parallelism. Since they are doing the same job they each need a bucket and brush. Otherwise they'll waste too much time walking back and forth. They also need to make sure that they paint all of the house and don't paint parts that have already been painted. An easy way to coordinate is to start at opposite ends of the house. Even if one person paints faster than the other, they will eventually meet and be done.
• Washing dishes is a little different. There is only one sink so two people can't wash at once. But one person can wash and another can dry. They form a short "pipeline" that dishes flow through one their way to be cleaned. Let's say it takes 30 seconds to wash a dish and 30 seconds to dry. 50 dishes would take 50 minutes for one person. For two people it would take 25 minutes and 30 seconds. 50 / 2 is 25, not 25.5. The extra half-minute comes from how pipelines work: the dryer can't start until the first dish has been washed.
• As pipelines get longer and more complex, managing delays becomes very important. Everyone has been to or seen videos of factories with their long conveyor belts, robots and people. If one section stops the whole line stops. Even tricker is managing stages that have different speeds. If you can make the candy faster than I can put them into boxes, then candy will build up behind me.
• To wash your clothes right you should separate the colors. A really fun way is to sit in the center of the room and toss darks, lights, and colors into different corners. A lot of high-tech companies gauge engineering candidates by their ability to solve the so-called Dutch National Flag problem. A lot of job candidates fail to figure out the trick, and perhaps fail the interview. I don't think that's a problem with the candidates or the basic idea of whiteboard interviews. I think it's a problem that we think DNP is a hard problem, when it is literally child's play.
• If you've ever seen an old war movie you've heard the characters reading numbers over the radio: One niner five alpha bravo ---repeat--- one niner five alpha bravo!. Numbers have to be clear so they say "niner" because "nine" might be mistaken for "five". They use words for letters so no one confuses B with D, P or E. They also repeat important information as an extra safety. It takes longer to communicate this way but getting it right is more important than efficiency. Computers communicate this way too. In fact the theory behind the tradeoff between efficiency and reliability, called Shannon's information theory, is one of the ideas that made modern communications possible. One of the neat things about coding theory is that you can control exactly how efficient or reliable you want something to be.

A book of How

There are a thousand of these little explanations one could come up with. Doubtless you've invented some of your own when learning a new thing or teaching it to someone else. Children do it all of the time, and we do it all of the time with children. But as far as I know they are not collected or connected with more general concepts. It's almost as though people are embarrassed to use such devices as a regular thing.

If you want to know What something is you look it up in the dictionary or encyclopedia. If you want to know Where something is you look at an atlas. When is covered by history books. Why and Who are kind of scattered about, but a lot of the answers are in history books and encyclopedias. All of these tools: encyclopedias, atlases, histories, were invented. They were invented and compiled one halting step at a time in response to need.

Computer science is, possibly, an attempt to write a book of How. We don't really know what it will look like, kids, but you're invited to help us find out.

The mental model we use to design programs assumes a "pointer" that has a definite position but occupies no space. The graphic casually called a pointer is just a marker of where this abstract pointer is at any given time. It's a pointer to a pointer, as it were. The pointer moves along a continuous path on the X,Y plane in response to the user manipulating some device in real space. In other words, your computer is an Etch-A-Sketch.

Fitts's Law (and more generally, Steering Law) has strongly influenced UI design. It states, more or less, that bigger targets are easier to hit. The cool thing is that it tells you exactly how much easier and what error rate you can expect for a given size and distance. It's one reason why high-value click targets like menu bars are placed at the edges of the screen: it gives them effectively infinite size because the edge stops the pointer for you. It also explains why deep hierarchical menus suck: the longer the path you have to steer the pointer through, the more likely it is you'll make a mistake.

Computers went through a twenty-year period when displays became larger and denser while pointing technology become more accurate, but didn't change the basic model. We could present more information at once; toolbars grew and sprouted palettes and ribbons and submenus. The presentation of content on screen became more and more faithful to the final output, under the rubric "what you see is what you get". Programmers came to assume that all of these trends would continue, and produced a host of guidelines founded on the little dot that follows an unbroken path:

• Don't make the user navigate paths that are too long, narrow, or complicated.
• Don't place common actions too far away from each other.
• "Content" goes in the center with menus, buttons, and toolbars arrayed around it.
• If you have more space, fill it with tools. If they can't see it they won't use it.
• If it's often used, put it near the edge of the screen.
• Keep the locations of things stable.
• Corners are ultra-prime locations.
• ...etc, etc, and so forth

WYSIWYT

Then cell phones mutated into real pocket computers. So far so good; we can dig into history and new research on how to deal with small screens. It's going to be a pain trying to fit all the information people now expect into a smaller area, but it's possible. But these small screens are also touchable. The union of what you see and what you touch breaks the pointer model and (partially) Steering Law: the "pointer" is no longer a point and its travel no longer continuous. After fifty years of Etch-A-Sketch we get to play with fingerpaint. All of the implications are not yet obvious, but here are six or seven:

Steering Law doesn't apply as strictly when the user can tap one point and then another without having to steer through a predetermined path in between. This is demonstrated by the new generation of iPad apps that use pop-out menus in ways that wouldn't work as well on a mouse-based system. The cost of a menu has gone down while the cost of screen real estate has gone up, producing a different solution.

Make touch areas more obvious, not less. The user has neither real tactile feedback nor a "hover" state to indicate that something will happen when he taps a particular place. Jacob Nielsen observes that iPad developers are currently partying like it's 1993, throwing all sorts of weird conventions into their apps while imitating print. Resist the temptation. A hallmark of new technology seems to be how it initially imitates or rejects its predecessors, then simulates them, and finally absorbs them. At the moment I'd say we're in between stage 1 and stage 2 with touchscreen interfaces. When you first get your hands on an e-reader with near-print resolution and "pages" you can flip with your fingers, it certainly feels like an apotheosis. Hey, you think, this is an acceptable simulation of a book. The illusion frays when you encounter a newspaper app that clings to some of the more annoying conventions of paper, does bizarre things when you tap a photo, and offers no search. [0]

The importance of large click targets goes way up on touch interfaces because we're using our big fat fingers instead of a geometric point. This brings up a fact we can no longer politely ignore: some fingers are fatter than others. Industrial designers know all about anthropometric variation. Anyone programming touch interfaces will have to, too -- or at least some, um, rules of thumb. A controlling factor is the average adult male finger width, about 2cm. Female hands, not to mention the hands of children, average smaller. In practice, touch buttons seem to be usable for most people down to 8 or 9 mm, but not much smaller. [1]

People with smaller hands are generally happier with the software keyboards on mobiles, leading some to speculate on a conspiracy of pointy-fingered elves. A friend of mine with very large hands has to hold it in his left and poke delicately at the keys with thumb and forefinger. [2] It's possible that one interface will not be able to accommodate everyone, and why not? We have different-sized mice and chairs and playing cards. It would be interesting to see how application designers experiment with configurable button sizes as they do with font sizes. Some software keyboards make common letters like E and T larger than the rest.

A related problem is how the finger and hand occlude other parts of the screen during interactions. Like those jerks in the front row of a movie theater, your hands get in the way at just the wrong time. Enlarging the thing being pressed is a good workaround, but what about the rest of the screen? There are quite a few first-person perspective games that overlay joystick controls on the main view. This kills peripheral vision: when turning left your hand covers most of the right side of the screen. Game developers know this, and that's why you often find several control schemes to choose from while they test out ideas. I suspect that those kinds of controls will gradually migrate to the bottom third of the screen.

More subtle changes are happening with eye focus. With a full-sized computer both eyes are focused on a single point about half a meter away. Pocket computers tend to be held closer and it's uncomfortable to close-focus for long periods. Also, the occlusion problem is partially solved by defocussing your eyes, so parts of the screen blocked for one eye are visible to the other. It's almost automatic. If you know a fast phone typer you can test this out by watching their eyes as they type, then blocking the screen with your finger. Their eyes will turn slightly inward as they change focus.

That feels somehow wrong to me. Great touchtyping is why people love hardware keyboards. A while ago I prototyped a Morse code "keyboard" for my mobile phone to see how it compared to software QWERTY keyboards. It sounds funny, and it is partly a joke, but it's also the minimum thing that could possibly work. With practice you can get fairly good.

One thing the Morse experiment taught me was that Fitts's Law didn't go away completely. This is not exactly a new revelation, but there is always some hot zone that is easiest to hit, unique to a given device size and orientation. On a pocket computer in portrait mode the hot zone is at the bottom. In landscape mode it's the area on either side, about two-thirds of the way up where the dit [.] and dah [-] buttons are. On a tablet the hot zone seems to be in the bottom right (or left) quadrant. Even so, the importance of the edges is less for touch than it is for mouse-based systems, because a virtual edge cannot stop your real finger.

A new interaction model may also need to take into account handedness and fatigue. Within minutes of using the first version of my "keyboard" I found an annoying bias towards dits in the Morse alphabet. On a telegraph the dit is three times faster than the dah so naturally Morse code uses them more. The surprising part was how uncoordinated my left hand was and how quickly it got tired. I ended up putting three buttons on each side to balance out the work and to reduce the number of taps per character.

And then you have multitouch, which knocks "mouse gestures" into a cocked hat, provided we can figure out how to use it effectively. Pinch/expand and rotate are very useful for controlling the "Z axis" perpendicular to the surface. There are also apps to simulate sound boards, pianos, and of course keyboards. Interestingly, multitouch doesn't break any design paradigms I can think of. It replaces a lot of them like resize and rotate "handles". Swipe can be (and is) overused but it's a natural replacement for pagination and scrolling. Using a two-finger tap as a replacement for "right click" to bring up context menus seems to be another home run. There are probably a lot of natural places for it as a modifier signal. For example, a drawing program might allow you to paint with two fingers and control brush size by the distance between them. It's not an unmixed blessing. Multitouch makes it harder to ignore accidental input from the palm and edges of the hand, which means the user can't treat a touch tablet as casually as paper just yet.

Touch interfaces remove one of the last physical barriers between users and digital data. Instead of manipulating data with a cartoon of a hand we control with silly instruments, we can poke it with our real fingers. This is deeply satisfying in a monkey kind of way. It gives programmers strange new problems and responsibilities. We will have to become amateur industrial designers just as we became amateur typographers, linguists, and psychologists. It puts us much closer, physically and emotionally, to the person on the other side of the glass. As users touch our programs, our programs are touching back.

Notes

[0] Stage 3 is when the new technology stops trying too hard to simulate older technology, and instead directly addresses (or renders moot) the underlying need, using its unique advantages. Stage 3 is usually gradual. My computer still calls its background layer "the Desktop", years after real desks as such (with drawers, pictures, clocks, inboxes, etc) distilled down to the humble table that holds my computer off the floor. Everything else has been sucked inside and transformed.

[1] This kind of stuff is fascinating, if you're the kind of person who's fascinated by this kind of stuff: "Thai females tended to have wider and thicker fingers but narrower knuckles than the females from Hong Kong, Britain, and India."
-- "Hand Anthropometry of Thai Female Industrial Workers", by N Saengchaiya and Y Bunterngchit, Journal of KMITNB, Vol 14, No 1, Jan 2004.

[2] Another friend of mine, who develops touchscreen apps for a living, tells me that finger size doesn't matter all that much. But I notice his fingers are rather pointy and elf-like...

Sweet Justice is a Javascript library you can drop onto any web page to create beautiful justified text. Even supercalifragilisticexpealadocious.

Sweet Justice lovingly inserts the obscure yet wonderful soft hyphen into the text of any element marked with the sweet-justice class, and turns on CSS text justification. It requires either jQuery or YUI3 to function.

Enjoy!
   - Carlos

From Wikipedia: Justification has been the preferred setting of type in many western languages through the history of movable type. This is due to the classic Western manuscript book page being built of a column or two columns, which is considered to look "best" if it is even-margined on the left and right. The classical Western column did not rigorously justify, but came as close as feasible when the skill of the penman and the character of the manuscript permitted. Historically, both scribal and typesetting traditions took advantage of abbreviations (sigla), ligatures, and swash to help maintain the rhythm and colour of a justified line.

The use of movable type solidified this preference from a technological point of view. It was much easier to handle and make emendations to large amounts of type that had words or syllables at the ends of lines than it was to respace the ends of lines.

Its use has only waned somewhat since the middle of the 20th century through the advocacy of the typographer Jan Tschichold's book Asymmetric Typography and the freer typographic treatment of the Bauhaus, Dada, and Russian constructivist movements.

Not all "flush left" settings in traditional typography were identical. In flush left text, words are separated on a line by the minimum word spacing built into the font.

Continuous casting typesetting systems such as the Linotype were able to reduce the jaggedness of the right-hand sides of adjacent lines of flush left composition by inserting self-adjusting space bands between words to evenly distribute white space, taking excessive space that would have occurred at the end of the line and redistributing it between words.

This feature, known as "ragged right" or "in and out ragged", was available in traditional dedicated typesetting systems but is absent from most if not all desktop publishing systems. Graphic designers and typesetters using desktop systems adjust word and letter spacing, or "tracking", on a manual line-by-line basis to achieve the same effect.

Two people can paint a house faster than one can. Honeybees work independently but pass messages to each other about conditions in the field. Many forms of concurrency, so obvious and natural in the real world, are actually pretty alien to the way we write programs today. It's much easier to write a program assuming that there is one processor, one memory space, sequential execution and a God's-eye view of the internal state. Language is a tool of thought as much as a means of expression, and the mindset embedded in the languages we use can get in the way. (Originally on Facebook's Engineering blog.)



The slovenliness of our language makes it easier for us to have foolish thoughts. The point is that the process is reversible.
-- George Orwell
Politics and the English Language

That language is an instrument of human reason, and not merely a medium for the expression of thought, is a truth generally admitted.
-- George Boole
The Laws of Thought

We're going through an inversion of scale in computing which is making parallelism and concurrency much more important. Single computers are no longer fast enough to handle the amounts of data we want to process. Even within one computer the relative speeds of processors, memory, storage, and network have diverged so much that they often spend more time waiting for data than doing things with it. The processor (and by extension, any program we write) is no longer a Wizard of Oz kind of character, sole arbiter of truth, at the center of everything. It's only one of many tiny bugs crawling over mountains of data.

Many hands make light work

A few years ago Tim Bray decided to find out where things stood. He put a computer on the internet which contained over 200 million lines of text in one very large file. Then he challenged programers to write a program to do some simple things with this file, such as finding the ten most common lines which matched certain patterns. To give you a feel for the simplicity of the task, Bray's example program employed one sequential thread of execution and had 78 lines of code, something you could hack up over lunch.

The computer was unusual for the time: it had 32 independent hardware threads which could execute simultaneously. The twist of the WideFinder challenge was that your program had to use all of those threads at once to speed up the task, while adding as little code as possible. The purpose was to demonstrate how good or bad everyday programming is at splitting large jobs into parallel tracks.

How hard could it be? I thought. Very hard, as it happened. I got up to 4 parallel processes before my program collapsed under its own weight. The crux of the problem was that the file was stored on a hard drive. If you've never peeked inside a hard drive, it's like a record player with a metal disc and a magnetic head instead of a needle. Just like a record it works best when you "play" it in sequence, and not so well if you keep moving the needle around. And of course it can only play one thing at a time. So I couldn't just split the file into 32 chunks and have each thread read a chunk simultaneously. One thread had to read from the file and then dole out parts of it to the others. It was like trying to get 31 housepainters to share the same bucket.

Parallelism is the act of taking a large job, splitting it up into smaller ones, and doing them at once. People often use "parallel" and "concurrent" interchangably, but there is a subtle difference. Concurrency is necessary for parallelism but not the other way around. If I alternate between cooking eggs and pancakes I'm doing both concurrently. If I'm cooking eggs while you are cooking pancakes, we are cooking concurrently and in parallel. Technically if I'm cooking eggs and you are mowing the lawn we are also working in parallel, but since no coordination is needed in that case there's nothing to talk about.

When I looked at other people's entries for hints I was struck by how almost all of them, good and bad, looked complicated and steampunky. Part of that was my unfamiliarity with the techniques, but another part was the lack of good support for parallelism which forced people to roll their own abstractions. (Ask four programmers to create a new abstraction and you'll get five and a half answers.) The pithiest entry was 130 lines of OCaml, a language that makes it somewhat easier to hack together "parallel I/O". Most of the others were several hundred lines long. Many people like me were not able to complete the challenge at all. If it's this difficult to parallelize a trivial string-counting program, what makes us think we're doing it right in complex ones?

Ideally, concurrency shouldn't leak into the logic of programs we're trying to write. Some really smart people would figure out the right way to do it. They would write papers with lots of equations in them and fly around to conferences for a few years until some other smart people figured out what the hell they were saying. Those people would go develop libraries in our favorite programming languages. Then we could just put import concurrent; at the top of our programs and be on our way. Concurrency would be another thing we no longer worry about unless we want to, like memory management. Unfortunately there is evidence that it won't be this clean and simple. A lot of things we take for granted may have to change. The switch to memory management wasn't all that easy either, come to think of it.

There are at least two concurrency problems to solve: how to get many components inside one computer to cooperate without stepping all over each other, and how to get many computers to cooperate without drowning in coordination overhead. These may be special cases of a more general problem and one solution will work for all. Or perhaps we'll have one kind of programming for the large and another for the small, just as the mechanics of life are different inside and outside of the cell.

At the far end of the spectrum are large distributed databases, such as those used by search engines, online retailers, and social networks. These things are enormous networks of computers that work together to handle thousands of writes and hundreds of thousands of reads every second. More machines in the system raises the odds that one of them will fail at any moment. There is also the chance that a link between groups of machines will fail, cutting the brain in half until it is repaired. There is a tricky balance between being able to read from such a system consistently and quickly and writing to it reliably. The situation is summed up by the CAP Theorem, which states that large systems have three desirable but conflicting properties: Consistency, Availability, and Partition-tolerance. You can only optimize for two at the expense of the third.

A Consistent/Available system means that reading and writing always works the way you expect, but requires a majority or quorum of nodes to be running in order to function. Think of a parliment that must have more than half of members present in order to hold a vote. If too many can't make it, say because a flood washes out the bridge, a quorum can't be formed and business can't proceed. But when enough members are in communication the decision-making process is fast and unambiguous. These categories are not rigidly exclusive. The status report problem is usually handled by having heirarchies of supervisors and employees aka "reports". The gossip consistency problem can be helped by tagging data with timestamps or version numbers so you can reconcile conflicting values. The interesting thing about Amazon's Dynamo is not eventual consistency but configurable consistency.

Consistent/Partitionable means that the system can recover from failures, but requires so much extra coordination that it collapses under heavy use. Imagine having to send and receive a status report for every decision made at your company. You'll always be current, and when you come back from vacation you will never miss a thing, but making actual progress would be very slow.

Available/Partitionable means that you can always read and write values, but the values you read might be out of date. A classic example is gossip: at any point you might not know the latest on what Judy said to Bill but eventually word gets around. When you have new gossip to share you only have to tell one or two people and trust that in time it will reach everyone who cares. Spreading gossip among computers is a bit more reliable because they are endlessly patient and (usually) don't garble messages.

After lots of groping around with billions of dollars of revenue at stake, people who build these large systems are coming to the conclusion that it's most important to always be able to write to a system quickly and read from it even in the face of temporary failures. Stale data is a consequence of looser coupling and greater autonomy needed to make that possible. It's uncomfortable to accept the idea that as the computing power of an Available/Partitionable system scales up, the fog of war descends on consistency, but in practice it's not the end of the world.

This was not a whimsical nor easy choice. Imagine Ebenezer Scrooge is making so much money that Bob Cratchit can't keep up. Scrooge needs more than one employee to receive and count it. To find out the grand total of his money at any point, he has to ask each of them for a subtotal. By the time Scrooge gets all the answers and adds them up, his employees have counted more money, and his total is already out of date. So he tells them to stop counting while he gathers subtotals. But this wastes valuable working time. And what if Scrooge adds another counting-house down the street? He'll have to pay a street boy, little Sammy Locke, to a) run to the other house and tell them to stop counting, b) gather their subtotals, c) deliver them to Scrooge, then d) run back to the other house to tell them to resume counting. What's worse, his customers can't pay him while this is happening. As his operation gets bigger Scrooge is faced with a growing tradeoff between stale information and halting everything to wait on Locke. If there's anything Scrooge likes less than old numbers, it's paying people to do nothing.

Scrooge's dilemma is forced upon him by basic physics. You can't avoid it by using electrons instead of street urchins. It's impossible for an event happening in one place (eg data changing inside one computer or process) to affect any other place (eg other computers or processes) until the information has had time to travel between them. Where those delays are small relative to performance requirements, Scrooge can get away with various forms of locking and enjoy the illusion of a shared, consistent memory space. But as programs spread out over more and more independent workers, the complexity needed to maintain that illusion begins to overwhelm everything else. This is not about speed-of-light effects or anything like that. I'm only talking about reference frames in the sense of "old news", such as when you find out your cousin had gotten married last year. Her wedding and your unawareness are both "true" relative to your reference frames until you receive news to the contrary.

Scratch that, reverse it

Shared memory can be pushed fairly far, however. Instead of explicit locks, Clojure and many newer languages use an interesting technique called software transactional memory. STM simulates a sort of post-hoc, fine-grained, implicit locking. Under this scheme semi-independent workers, called threads, read and write to a shared memory space as though they were alone. The system keeps a log of what they have read and written. When a thread is finished the system verifies that no data it read was changed by any other. If so the changes are committed. If there is a conflict the transaction is aborted, changes are rolled back and the thread's job is retried. While threads operate on non-overlapping parts of memory, or even non-overlapping parts of the same data structures, they can do whatever they want without the overhead of locking. In essence, transactional memory allows threads to ask for forgiveness instead of permission.

As you might have guessed from those jolly hints about conflict and rollback, STM has its own special problems, like how to perform those abort/retry cycles efficiently on thousands of threads. It's fun to imagine pathological conflict scenarios in which long chains of transactions unravel like a cheap sweater. Also, STM can only handle actions that are undoable. You can't retry most kinds of I/O for the same reason you can't rewind a live concert. This is handled by queueing up any non-reversible actions, performing them outside of the transaction, caching the result in a buffer, and replaying as necessary. Read that sentence again. There is a recent paper about an interesting variation on this theme called HyTM, which appears to do a copy-on-write instead of performing writes to shared memory.

Hold this thread as I walk away

Undeniably awesome and clever as STM threads are, I'm not convinced that shared memory makes sense outside of the "cell membrane" of a single computer. Throughput and latency always have the last laugh. A concurrent system is fundamentally limited by how often processes have to coordinate and the time it takes them to do so. As of this writing computer memory can be accessed in about 100 nanoseconds. Local network's latency is measured in microseconds to milliseconds. Schemes that work well at local memory speeds don't fly over a channel one thousand times slower. Throughput is a problem too: memory can have one hundred times the throughput of network, and is shared among at most a few dozen threads. A large distributed database can have tens of thousands of independent threads contending for the same bandwidth.

If we can't carry the shared-memory model outside of the computer, is there some other model we can bring inside? Are threads, ie semi-independent workers that play inside a shared memory space, absolutely necessary? In his "standard lecture" on threads Xavier Leroy details three reasons people use them:

• Shared-memory parallelism using locks or transactions. This is explicitly disowned in both Erlang and Leroy's OCaml in favor of message-passing. His argument is that it's too complex, especially in garbage-collected languages, and doesn't scale. (This was in 2002 when multiprocessors were rare and strange beasts.)
• Overlapping I/O and computation, ie while thread A is waiting for data to be sent or received, threads B-Z can continue their work. Overlapping (aka non-blocking I/O) is needed to solve problems like WideFinder efficiently. This is often thwarted by low-level facilities inside the operating system that were written without regard to parallelism. Leroy thinks this should be fixed at the OS level instead of making every program solve it again and again.
• Coroutines, which allow different functions to call each other repeatedly without generating an infinitely long stack of references back to the first call. This looks suspiciously like message-passing.

Message-passing, which first appeared in Smalltalk, is the core abstraction of Joe Armstrong's programming language Erlang. Erlang programs do things that make programmers take notice, like run some of the busiest telephone switches for years without fail. It approaches concurrency with three iron rules: no shared memory even between processes on the same computer, a standard format for messages passed between processes, and a guarantee that messages are read in the order in which they were received. The first rule is meant to avoid the heartaches described above and embraces local knowledge over global state. The second and third keep programmers from endlessly reinventing schemes for passing messages between processes. Every Erlang process has sovereign control over its own memory space and can only affect others by sending well-formed messages. It's an elegant model and happens to be a cleaned-up version of the way the internet itself is constructed. Message-passing is already one of the axioms of concurrent distributed computation, and may well be universal.

A lot of writeups repeat a "nine nines", ie 99.9999999% reliability claim for Erlang-based Ericsson telephone switches owned by British Telecoms. This works out to 31 milliseconds of downtime per year, which hovers near the edge of measurability, not to say plausibility. I was present at a talk Armstrong gave in early 2010 during which he was asked about this. There was a little foot shuffling as he qualified it: it was actually 6 or so seconds of downtime in one device during a code update. Since BT had X devices over Y years, they calculated it as 31ms of average downtime per device per year. Or something like that. Either way it's an impressive feat.

There are probably more axioms to discover. Languages become more powerful as abstractions are made explicit and standardized. Message-passing says nothing about optimizing for locality, ie making sure that processes talk with other processes that are located nearby instead of at random. It might be cool to have a standard way to measure the locality of a function call. Languages become even more powerful when abstractions are made first-class entities. For example, languages that can pass functions as arguments to other functions can generate new types of higher-order functions without the programmer having to code them by hand. A big part of distributed computing is designing good protocols. I know of no language that allows protocols as first-class entities that can be passed around and manipulated like functions and objects are. I'm not even sure what that would look like but it might be interesting to try out.

There is a lot of sound and fury around parallelism and concurrency. I don't know what the answer will be. I personally suspect that a relaxed, shared-memory model will work well enough within the confines of one computer, in the way that Newtonian physics works well enough at certain scales. A more austere model will be needed for a small network of computers, and so on as you grow. Or perhaps there's something out there that will make all this lockwork moot.

Suppose that breakthroughs, like the phoenix, do keep their own schedules. We can still examine their nesting sites for clues. Some breakthroughs happen when the demand on a technology overwhelms regular innovation's ability to deliver. This happens in a big way during wartime and on a smaller scale every day. When I discover some new way to look at a program I'm writing, it's only after I've tried to stuff up the cracks with hasty code and deranged notes to myself. Maybe that's a formula: look for vital technologies that seem overly complex and burdened with new words.

"Overly complex" and "burdened with words" are hallmarks of the faux-historical Steampunk style and the predigital technology it cariacatures. I think we are especially fascinated by Steampunk machines because the mechanisms of control are obvious and human-scaled. This lever goes like so, the cable pulls the thingabob thusly, causing the flywheel to spin, etc. Like all cariacatures, Steampunk aesthetic exaggerates the features that make its subject distinctive. Behind the brass fittings and oddly-named components is a portrait of the mind which thought them necessary: ingenious mechanical design combined with astonishing ignorance about the principles of process control those designs embody. There is real tension there, a naive insistence that if something is important it must also be big enough to see.

Steampunk is ever with us

As a wee lad in the last century I had a job repairing video cassette recording devices. Dozens of parts had to be properly greased, timed, or roughened up to get the right amount of tension and smooth unwinding. They had names like the Reverse/Forward Idler, Flywheel Belt, Pinch Roller, and Flying Erase Head. There was a long coil of wire whose only purpose was to delay the luma signal just enough to sync it with the chroma. Really old VCRs are as Steampunk as they come. [0]

Over time VCRs became smaller and vastly less complex while performing the same tasks as their ancestors. Very clever people spent decades improving the clockwork of video processing. Now of course we know the real breakthrough lay in getting rid of the clockwork.

It's probably a mistake to feel smug about that. The future isn't measured by years or technology, but by how it disagrees with the past about what is and what is not important. For example, I suspect the future will mock our assumption that computer intelligence will in any way resemble human intelligence, for the same reason we mock brass-handled Victorian contraptions. The idea is so central yet unfounded that it's bound to be wrong somehow. Whatever it is we're grasping toward, it may be "artificial intelligence" only in the sense that a car is an "iron horse".

In the first of the famous SICP lectures Hal Ableson explains that we are as ignorant about the true nature of computing as early Egyptians were about geometry. Those guys did impressive stuff with string and rocks but they didn't really understand why it worked. It took Euclid to distill the axioms and methods we now file under geometry. The misfit name geo-metry, ie earth-measurement, is a fossil of its confused history. Perhaps the jumbled bag of tricks we call "computer science" —not a science, and not about computers— is similarly waiting for something to hatch.

Notes

[0] My father disagrees with this interpretation. He points out that in the 1970s consumer video players were a huge gamble. It was decided that they had to be super reliable, and so were overbuilt. The evolution of VCRs post 1980 is a story of materials science (eg Teflon-coated plastic parts instead of greased metal and rubber belts) and amazing advances in integrated circuits. In short, VCRs became less reliable at roughly the same rate they became cheap enough that it didn't matter.

He has a point. The designers of early VCRs were not confused or stupid. Nevertheless they produced the most complex mechanical devices ever seen in the home, the maintainence of which provided employment for thousands.

Also see the Ampex VCR from 1956, which was the size of a washer and dryer, and the failed SelectaVision video disc, which played video encoded onto 12-inch plastic records.

Update: Check out its new home at jsrosettastone.com

In late 2009 I was porting a largish prototype from jQuery to YUI3 and kept wishing for a cheatsheet to help me translate the idioms of one library to the other. A few months and lots of expert help later, here is a Rosetta Stone for jQuery and YUI3. You can read and comment on the HTML version, download the PDF, or fork the project on GitHub. It is by no means complete or authoritative but I hope it's useful.

This is a reference, not an evaluation or comparison of the libraries. If you have an opinion about which one is "better", that's great, but I don't want to hear it.

jQuery and YUI3 only overlap in certain areas like cross-browser DOM manipulation, CSS, Ajax, etc. In general I think jQuery's convenience methods, core selector support, and method chaining are its greatest strengths. YUI3 also has selectors and chaining, but favors configurability, breadth, and support for complex applications rather than terseness.

Enjoy!

HTML version, PDF Version

Any savvy web developer can tell you how many kilobytes their code consumes. They bundle, minify, compress and tune the data sent out to within an inch of its life. Wire weight is easy to measure and has a direct impact on your application's launch time. But how many milliseconds does it take the user's computer to parse and load your code once it's arrived? What differences are there between CPUs, operating systems, browsers and plugins? What speed leaks are we overlooking?

• The Test
• The Libraries
• Results
• Minification
• So What?
• Try this at home
• Appendix
  • The Curious Case of Chrome
  • Debugging the Benchmark
  • Adventures in Garbage Collecting
  • A Tragic Opera (updated)

As you look at the data below, keep in mind four things:

1. Your code is not the only code running on the user's computer
2. Parse-n-load time comes down to available CPU cycles and RAM
3. The fastest CPUs aren't getting much faster
4. The average consumer CPU is getting slower

Even if you look at just sexy new hardware it's hard to ignore low-power clients: there are about 50 million netbooks and 43 million iPhones out there, alongside 10-15 million Android devices. Almost all of them are in the 600-1,600 MHz range and have less than 512MB of RAM. A juicy, growing slice of the market wants to use your software on the equivalent of a desktop from 1998. This includes rich Westerners as well as people in the fastest-growing international markets.

The test harness loads a given block of Javascript from a local file over and over and measures the setup time. My test subjects were the Yahoo User Interface (YUI) libraries, Scriptaculous, and jQuery UI. I've also included the main Javascript application code from GitHub.

The core test is as simple as can be: record a start time, load the script, record the end time, and repeat over 1,000 iterations. The tests were run on recently-booted machines with no other programs running. You can check out the project on GitHub and play along at home.

Flickr: charlietakesphotos

• YUI 2.8.0r4, 390KB
  partial (dom, event, datasource, datatable, layout, tabview, treeview, menu)
• YUI 3.0 build 1549, 311KB
  "kitchen sink"
• Scriptaculous 1.8.3, 159KB
  "kitchen sink"
• jQuery UI 1.7.2, 359KB
  "kitchen sink", minus translations
• GitHub.com 09 Feb 2010, 211KB
  main application code, including jQuery 1.4

The point of this benchmark is to compare browsers and CPUs. Comparing the parse-n-load of different libraries puts you on shaky ground. For example, the YUI2 libraries are as of this writing more comprehensive than, say, YUI3 or Scriptaculous. On the other hand it's rare for an application to load every module as we're doing here. Also, each library has a diffferent approach to initialization. YUI2 does a lot of work up-front while YUI3 does things more lazily. From there you get into a complex question about what benefits each library buys you. Do not base the choice of library on this benchmark.

Presario R3000 1.6GHz, WinXP SP3

MacBook Air 1.6GHz, OSX 10.5.8

NB: the MacBook Air had overheating problems so some of these numbers in this table may be skewed upwards. On the other hand, that's precisely the kind of crap your users have to deal with.

So it seems that there is a measurable cost to parse-n-load, and parsing speed does not seem to be correlated with the speed of the interpreter or DOM. Chrome has some anomalies in the data so for now I am withholding judgement. See "Curious Case of Chrome" in the appendix.

There is a noticeable spread between different browsers on the same hardware and OS. Firefox 3.5 is a few points slower than 3.0, but 3.6 improved on that. Internet Explorer is surprisingly fast at parse-n-load across all tested versions. I didn't include standard deviations because aside from some pathological cases they were small. If you run the benchmark for yourself you will get mean average, stddev, and a time series graph for your enjoyment.

Here is a comparison of the YUI libraries in "raw" form (with comments, whitespace, etc) and the same code minified using YUI Compressor. As expected, minification helps parse-n-load in addition to network transmission time. This is probably due to the absence of comments and extra whitespace.

If you have a large amount of Javascript in your application it's natural to bundle it all up into one file to save on network transit time and increase cacheability. But if certain parts of your application only use certain parts of the bundle, you might cause the user to unnecessarily parse unused code on every page load. The ideal would be to decouple three things that are now tightly wound together: network transit, parse-n-load, and cacheability. There is a lot of work going on in this space but so far there is no silver bullet.

• Alexander Limi from Mozilla has a proposal to use zip files for bundling.
• The folks at 280North have found a neat way to do multi-file bundling in their Cappuccino framework, using existing technology.
• Google is proposing an extension to HTTPS that allows multiple concurrent streams over a single TCP connection.
• SproutCore and the Google Mobile Team recently demonstrated ways to load Javascript code as dumb strings that is evaluated at a time of the programmer's choosing.

The Parse-N-Load benchmark is open source and free for use. It's early and I'm sure there are bugs. If you have other kinds of hardware (Netbooks! Windows! Linux!), please try it out and let me know what you find.

Appendix

While Google Chrome appears to be an order of magnitude faster at parse-n-load, the truth may be a little more complex. Running this benchmark in Chrome sometimes produces sharp cliffs in the time series graph, especially on slower CPUs. That might be the V8 engine's inline caching kicking in. I also suspect it could be caching the machine code it compiles on the first pass. Or this could be something silly like the CPU coming out of low-power mode. If anyone who knows more about what's going on can speak up, please do.

Figure 0: Chrome's V8 Javascript engine has two speeds: fast and very fast.

The first problem that came up was different blocking behavior between browsers. In Safari 4 (but not 3) if you create a script tag that points to an external file, that action will block, ie, wait until that file is completely parsed and loaded. This makes timing it very easy. In Firefox, however, this action asynchronous: the statement that creates the script element returns immediately and the file is loaded in a separate thread. This means you have to set up a callback in the separate thread to both measure elapsed time and kick off the next iteration of the test. You have to be careful not the blow the stack with too many nested function calls. Google Chrome is also asynchronous and has an altogether different stack behavior. If all that wasn't enough, browsers have very different memory allocation behavior, of which more (oh, much more) below.

Every browser seems to have a different system for allocating memory while parsing Javascript code. When you graph the results from Safari 4 this is what I saw initially:

Figure 1: the effects of GC halts

Interesting. All of the source files in the benchmark are local so that's not I/O wait. The regularity of huge spikes suggests that the browser is pausing every so often to free up memory via garbage collection. When you remove those spikes another interesting pattern shows up. Here is the same graph with the top 5% of datapoints removed:

Figure 2: 95th percentile graph of Javascript load times in Safari 4

It appears that the parse-n-load time of a given piece of Javascript in Safari 4 will increase linearly with the amount of garbage. The load time can grow as much as 3X longer than normal before GC kicks in. I'm not certain whether this is an artifact of the benchmark or if it actually happens a lot during real-world use. When I added code to explicitly delete the previous script tag before creating a new one, the sawtooth elongated but did not go away. Other browsers exhibit similar halt-the-world GC behavior but only Safari 4 and Opera 10.5alpha have this sawtooth. Firefox's graph stays fairly horizontal but has many more small spikes:

Figure 3: 95th percentile graph of Javascript load times in Firefox 3.5.3

Initially results for Opera 10.0 had asterisks because I couldn't get it to run the complete test. Opera got steadily slower up to 250 iterations after which it started serious thrashing and had to be killed. Geoffrey Sneddon from Opera software kindly suggested an alternate way to run this benchmark, ie destroying and re-creating the iframe for each trial instead of overwriting the same document object. This new method (dubbed "v2") works well and the Opera column has been updated. New test runs against the other browsers did not move their 95th percentile numbers more than a few percent, though the standard deviations and number of GC pauses did decrease.

Geoffrey also mentioned that while the condition I triggered is rare in the wild, they will work to fix it in a future release.

Figure 4: Opera 10 choking on the parse-n-load benchmark v1

Figure 5: Opera 10 doing nicely with benchmark v2

Safari 3 hit the wall even earlier, triggering an Out-of-Memory error after just a couple hundred iterations. The v2 of the benchmark works much better.

Figure 6: with v1 Safari 3 would throw an OOM at this point.

Figure 6: Safari 3 likes benchmark v2 much better.

]]> tag:org.bueno.carlos.0d6643bc1ea2676cd7e009291b767dfea04739ef 2010-02-01T12:00:00.000-08:00 2010-02-01T12:00:00.000-08:00 23 February 2010

Another generation of technology has passed and Unicode support is almost everywhere. The next step is to write software that is not just “internationalized” but truly multilingual. In this article we will skip through a bit of history and theory, then illustrate a neat hack called accent-folding. Accent-folding has its limitations but it can help make some important yet overlooked user interactions work better.
Illustration by Kevin Cornell
(Originally appeared on A List Apart.)

A common assumption about internationalization is that every user fits into a single locale like “English, United States” or “French, France.” It’s a hangover from the PC days when just getting the computer to display the right squiggly bits was a big deal. One byte equaled one character, no exceptions, and you could only load one language’s alphabet at a time. This was fine because it was better than nothing, and because users spent most of their time with documents they or their coworkers produced themselves.

Today users deal with data from everywhere, in multiple languages and locales, all the time. The locale I prefer is only loosely correlated with the locales I expect applications to process.

Consider this address book:

• Fulanito López
• Erik Lørgensen
• Lorena Smith
• James Lö

If I compose a new message and type “lo” in the To: field, what should happen? In many applications only Lorena will show up. These applications “support Unicode,” in the sense that they don’t corrupt or barf on it, but that’s all.
Fig 1. Hey Entourage, where are my contacts?

This problem is not just in address books. Think about inboxes, social bookmarks, comment feeds, users who speak multiple languages, users in internet cafés in foreign countries, even URLs. Look at the journalist Ryszard Kapuściński and how different websites handle his name:

• Wikipedia: Ryszard Kapuściński (canonical URL)
• Wikipedia: Ryszard Kapuscinski (hand-coded alternate)
• Wikipedia: Ryszard Kapusciński (not found)
• Wikipedia: Rÿszarḋ Kåpuścińsḳi (not found)
• Spock: Rÿszarḋ Kåpuścińsḳi (accent-folded, redirects to canonical URL)

There is no excuse for your software to play dumb when the user types “cafe” instead of “café.”

Áçčềñṭ-Ḟøłɖǐṅg

In specific applications of search that favor recall over precision, such as our address book example, á, a, å, and â can be treated as equivalent. Accents (a.k.a diacritical marks) are pronunciation hints that don’t affect the textual meaning. Entering them can be cumbersome, especially on mobile devices.
Fig 2. accent-folding in an autosuggest widget

An accent-folding function essentially maps Unicode characters to ASCII equivalents. Anywhere you apply case-folding, you should consider accent-folding, and for exactly the same reasons. With accent-folding, it doesn’t matter whether users search for cafe, café or even çåFé; the results will be the same.

Be aware that there are a million caveats to accent rules. You will almost certainly get it wrong for somebody, somewhere. Nearly every alphabet has a few extra-special marks that do affect meaning, and, of course, non-Western alphabets have completely different rules.

A minor gotcha is the Unicode "fullwidth" Roman alphabet. These are fixed-width versions of plain ASCII characters designed to line up with Chinese/Japanese/Korean characters (e.g., "１９７９年８月１５日"). They reside in 0xff00 to 0xff5e and should be treated as equivalent to their ASCII counterparts.

Hey man, I’m only here for the copy/paste

I’ve posted more complete examples on GitHub, but for illustration, here’s a basic accent-folder in Javascript:

var accentMap = {
  'á':'a', 'é':'e', 'í':'i','ó':'o','ú':'u'
};

function accent_fold (s) {
  if (!s) { return ''; }
  var ret = '';
  for (var i = 0; i < s.length; i++) {
    ret += accent_map[s.charAt(i)] || s.charAt(i);
  }
  return ret;
};

Regular Expressions

Regular expressions are very tricky to make accent-aware. Notice that in Fig. 2 only the unaccented entries are in bold type. The problem is that the Unicode character layout does not lend itself to patterns that cut across languages. The proper regex for “lo” would be something insane like:

[LlĹ弾ĻļḶḷḸḹḼḽḺḻŁłŁłĿŀȽƚɫ][OoÓóÒòŎŏÔôỐốỒồỖöȪȫŐőÕõṌȭȮȯǾǿ...ǬǭŌ]

Never, never do this. As of this writing, few regular expression engines support shortcuts for Unicode character classes. PCRE and Java seem to be in the vanguard. You probably shouldn’t push it. Instead, try highlighting an accent-folded version of the string, and then use those character positions to highlight the original, like so:

// accent_folded_hilite("Fulanilo López", 'lo')
//   --> "Fulani<b>lo</b> <b>Ló</b>pez"
//
function accent_folded_hilite(str, q) {
  var str_folded = accent_fold(str).toLowerCase().replace(/[<>]+/g, '');
  var q_folded = accent_fold(q).toLowerCase().replace(/[<>]+/g, '');

  // Create an intermediate string with hilite hints
  // Example: fulani<lo> <lo>pez
  var re = new RegExp(q_folded, 'g');
  var hilite_hints = str_folded.replace(re, '<'+q_folded+'>');

  // Index pointer for the original string
  var spos = 0;
  // Accumulator for our final string
  var highlighted = '';

  // Walk down the original string and the hilite hint
  // string in parallel. When you encounter a < or > hint,
  // append the opening / closing tag in our final string.
  // If the current char is not a hint, append the equiv.
  // char from the original string to our final string and
  // advance the original string's pointer.
  for (var i = 0; i< hilite_hints.length; i++) {
    var c = str.charAt(spos);
    var h = hilite_hints.charAt(i);
    if (h === '<') {
      highlighted += '<b>';
    } else if (h === '>') {
      highlighted += '</b>';
    } else {
      spos += 1;
      highlighted += c;
    }
  }
  return highlighted;
}

The previous example is probably too simplistic for production code. You can’t highlight multiple terms, for example. Some special characters might expand to two characters, such as “æ” --> “ae” which will screw up spos. It also strips out angle-brackets (<>) in the original string. But it’s good enough for a first pass.

Accent-folding in YUI Autocomplete

YUI's Autocomplete library has many hooks and options to play with. Today we’ll look at two overrideable methods: filterResults() and formatMatch(). The filterResults method allows you to write your own matching function. The formatMatch method allows you to change the HTML of an entry in the list of suggested matches. You can also download a complete, working example with all of the source code.

<!-- this is important to tell javascript to treat
the strings as UTF-8 -->
<meta http-equiv="content-type" content="text/html;charset=utf-8" />

<!-- YUI stylesheets -->
<link rel="stylesheet" type="text/css"
 href="http://yui.yahooapis.com/2.7.0/build/fonts/fonts-min.css" />
<link rel="stylesheet" type="text/css"
 href="http://yui.yahooapis.com/2.7.0/build/autocomplete/assets/skins/sam/autocomplete.css" />

<!-- YUI libraries: events, datasource and autocomplete -->
<script type="text/javascript"
 src="http://yui.yahooapis.com/2.7.0/build/yahoo-dom-event/yahoo-dom-event.js"></script>
<script type="text/javascript"
 src="http://yui.yahooapis.com/2.7.0/build/datasource/datasource-min.js"></script>
<script type="text/javascript"
 src="http://yui.yahooapis.com/2.7.0/build/autocomplete/autocomplete-min.js"></script>

<!-- contains accent_fold() and accent_folded_hilite() -->
<script type="text/javascript" src="accent-fold.js"></script>

<!-- Give <body> the YUI "skin" -->
<body class="yui-skin-sam">
  <b>To:</b>
  <div style="width:25em">

    <!-- Our to: field -->
    <input id="to" type="text" />

    <!-- An empty <div> to contain the autocomplete -->
    <div id="ac"></div>

  </div>
</body>

<script>

// Our static address book as a list of hash tables
var addressBook = [
  {name:'Fulanito López', email:'fulanito@example.com'},
  {name:'Erik Lørgensen', email:'erik@example.com'},
  {name:'Lorena Smith',   email:'lorena@example.com'},
  {name:'James Lö',       email:'james@example.com'}
];

/*
Iterate our address book and add a new field to each
row called "search." This contains an accent-folded
version of the "name" field.
*/
for (var i = 0; i< addressBook.length; i++) {
  addressBook[i]['search'] = accent_fold(addressBook[i]['name']);
}

// Create a YUI datasource object from our raw address book
var datasource = new YAHOO.util.LocalDataSource(addressBook);

/*
A datasource is tabular, but our array of hash tables has no
concept of column order. So explicitly tell the datasource
what order to put the columns in.
*/
datasource.responseSchema = {fields : ["email", "name", "search"]};

/*
Instantiate the autocomplete widget with a reference to the
input field, the empty div, and the datasource object.
*/
var autocomp = new YAHOO.widget.AutoComplete("to", "ac", datasource);

// Allow multiple entries by specifying space
// and comma as delimiters
autocomp.delimChar = [","," "];

/*
Add a new filterResults() method to the autocomplete object:
Iterate over the datasource and search for q inside the
"search" field. This method is called each time the user
types a new character into the input field.
*/
autocomp.filterResults = function(q, entries, resultObj, cb) {
    var matches = [];
    var re = new RegExp('\\b'+accent_fold(q), 'i');
    for (var i = 0; i < entries.length; i++) {
        if (re.test(entries[i]['search'])) {
            matches.push(entries[i]);
        }
    }
    resultObj.results = matches;
    return resultObj;
};

/*
Add a new formatResult() method. It is called on each result
returned from filterResults(). It outputs a pretty HTML
representation of the match. In this method we run the
accent-folded highlight function over the name and email.
*/
autocomp.formatResult = function (entry, q, match) {
    var name = accent_folded_hilite(entry[1], q);
    var email = accent_folded_hilite(entry[0], q);
    return name + ' <' + email + '>';
};

//fin
</script>

About those million caveats...

This accent-folding trick works primarily for Western European text, but it won’t work for all of it. It exploits specific quirks of the language family and the limited problem domains of our examples, where it’s better to get more results than no results. In German, Ü should probably map to Ue instead of just U. A French person searching the web for thé (tea) would be upset if flooded with irrelevant English text.

You can only push a simple character map so far. It would be very tricky to reconcile the Roman “Marc Chagall” with the Cyrillic “Марк Шагал” or Hebrew “מאַרק שאַגאַל.” There are very interesting similarities in the characters but a magical context-free two-way mapping is probably not possible.

On top of all that there is another problem: One language can have more than one writing system. Transliteration is writing a language in a different alphabet. It’s not quite the same as transcription, which maps sounds as in “hola, que paso” --> “oh-la, keh pah-so.” Transliterations try to map the written symbols to another alphabet, ideally in a way that’s reversible.

These four sentences all say “Children like to watch television” in Japanese:

• One Way: 子供はテレビを見るのが好きです。
• Another Way: こども は てれび を みる の が すき です 。
• Romaji: kodomo wa terebi o miru noga suki desu.
• Cyrillic: кодомо ва тэрэби о миру нога суки дэсу.

For centuries people have been inventing ways to write different languages with whatever keyboards or typesetting they had available. So even if the user reads only one language, they might do so in multiple transliteration schemes. Some schemes are logical and academic, but often they are messy organic things that depend on regional accent and historical cruft. The computer era kicked off a new explosion of systems as people learned to chat and send emails in plain ASCII.

There is a lot of prior work on this problem and two ready paths you can choose: The right way and the maybe-good-enough way. Neither have the simplicity of our naïve hash table, but they will be less disappointing for your users in general applications.

The first is International Components for Unicode (ICU), a project that originated in the early nineties at Taligent. It aims to be a complete, language-aware transliteration, Unicode, formatting, everything library. It’s big, it’s C++/Java, and it requires contextual knowledge of the inputs and outputs to work.

The second is Unidecode, a context-free transliteration library available for Perl and Python. It tries to transliterate all Unicode characters to a basic Latin alphabet. It makes little attempt to be reversible, language-specific, or even generally correct; it’s a quick-and-dirty hack that nonetheless is pretty comprehensive.

Accent-folding in the right places saves your users time and makes your software smarter. The strict segregation of languages and locales is partially an artifact of technology limitations which no longer hold. It’s up to you how far you want to take things, but even a little bit of effort goes a long way. Good luck!

Italian Translation